[==========] Running 3 tests from 1 test case. [----------] Global test environment set-up. [----------] 3 tests from CreateTableITest [ RUN ] CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation I1010 20:58:34.083943 1686 external_mini_cluster.cc:586] Running /tmp/run_tha_testXiwMYW/build/release/bin/kudu-master kudu-master --fs_wal_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/master-0 --fs_data_dirs=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/master-0 --rpc_bind_addresses=127.0.0.1:0 --webserver_interface=localhost --webserver_port=0 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/master-0 --tablet_creation_timeout_ms=1000 --server_dump_info_path=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/master-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 --unlock_experimental_flags --unlock_unsafe_flags WARNING: Logging before InitGoogleLogging() is written to STDERR W1010 20:58:34.088894 1691 flags.cc:294] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true I1010 20:58:34.089530 1691 mem_tracker.cc:140] MemTracker: hard memory limit is 11.777661 GB I1010 20:58:34.089543 1691 mem_tracker.cc:142] MemTracker: soft memory limit is 7.066597 GB I1010 20:58:34.090481 1691 master_main.cc:58] Initializing master server... I1010 20:58:34.090622 1691 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 968115 I1010 20:58:34.091334 1691 server_base.cc:167] Could not load existing FS layout: Not found: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/master-0/instance: No such file or directory (error 2) I1010 20:58:34.091346 1691 server_base.cc:168] Creating new FS layout I1010 20:58:34.291821 1691 fs_manager.cc:354] Generated new instance metadata in path /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/master-0/instance: uuid: "d4cf8c406c3b4693b9535068d9267967" format_stamp: "Formatted at 2016-10-10 20:58:34 on dist-test-slave-dist-test-slave-ymsa" I1010 20:58:34.297667 1691 fs_manager.cc:243] Opened local filesystem: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/master-0 uuid: "d4cf8c406c3b4693b9535068d9267967" format_stamp: "Formatted at 2016-10-10 20:58:34 on dist-test-slave-dist-test-slave-ymsa" I1010 20:58:34.300099 1691 master_main.cc:61] Starting Master server... I1010 20:58:34.303762 1691 rpc_server.cc:164] RPC server started. Bound to: 127.0.0.1:46501 I1010 20:58:34.303809 1691 webserver.cc:125] Starting webserver on localhost:0 I1010 20:58:34.303817 1691 webserver.cc:136] Document root disabled I1010 20:58:34.304065 1691 net_util.cc:195] Address 127.0.0.1:0 for localhost:0 duplicates an earlier resolved entry. I1010 20:58:34.304234 1691 webserver.cc:220] Webserver started. Bound to: http://127.0.0.1:33610/ I1010 20:58:34.304344 1691 server_base.cc:239] Dumped server information to /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/master-0/info.pb I1010 20:58:34.309262 1745 tablet_bootstrap.cc:380] T 00000000000000000000000000000000 P d4cf8c406c3b4693b9535068d9267967: Bootstrap starting. I1010 20:58:34.309408 1745 tablet_bootstrap.cc:542] T 00000000000000000000000000000000 P d4cf8c406c3b4693b9535068d9267967: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:34.309495 1745 tablet_bootstrap.cc:482] T 00000000000000000000000000000000 P d4cf8c406c3b4693b9535068d9267967: No blocks or log segments found. Creating new log. I1010 20:58:34.309602 1745 log.cc:347] Log is configured to *not* fsync() on all Append() calls I1010 20:58:34.310008 1745 tablet_bootstrap.cc:380] T 00000000000000000000000000000000 P d4cf8c406c3b4693b9535068d9267967: No bootstrap required, opened a new log I1010 20:58:34.310313 1745 raft_consensus.cc:275] T 00000000000000000000000000000000 P d4cf8c406c3b4693b9535068d9267967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4cf8c406c3b4693b9535068d9267967" member_type: VOTER } I1010 20:58:34.310335 1745 raft_consensus.cc:306] T 00000000000000000000000000000000 P d4cf8c406c3b4693b9535068d9267967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:34.310343 1745 raft_consensus.cc:502] T 00000000000000000000000000000000 P d4cf8c406c3b4693b9535068d9267967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4cf8c406c3b4693b9535068d9267967, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:34.310349 1745 consensus_queue.cc:168] T 00000000000000000000000000000000 P d4cf8c406c3b4693b9535068d9267967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:34.310353 1745 raft_consensus.cc:319] T 00000000000000000000000000000000 P d4cf8c406c3b4693b9535068d9267967 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I1010 20:58:34.310356 1745 raft_consensus.cc:375] T 00000000000000000000000000000000 P d4cf8c406c3b4693b9535068d9267967 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:34.310359 1745 raft_consensus.cc:2031] T 00000000000000000000000000000000 P d4cf8c406c3b4693b9535068d9267967 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:34.310520 1745 raft_consensus.cc:1981] T 00000000000000000000000000000000 P d4cf8c406c3b4693b9535068d9267967 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.512s I1010 20:58:34.310540 1745 raft_consensus.cc:391] T 00000000000000000000000000000000 P d4cf8c406c3b4693b9535068d9267967 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4cf8c406c3b4693b9535068d9267967" member_type: VOTER } I1010 20:58:34.312047 1686 external_mini_cluster.cc:622] Started /tmp/run_tha_testXiwMYW/build/release/bin/kudu-master as pid 1691 I1010 20:58:34.312145 1686 external_mini_cluster.cc:586] Running /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-0 --fs_data_dirs=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-0 --rpc_bind_addresses=127.6.150.0:0 --local_ip_for_outbound_sockets=127.6.150.0 --webserver_interface=127.6.150.0 --webserver_port=0 --tserver_master_addrs=127.0.0.1:46501 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-0 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 --unlock_experimental_flags --unlock_unsafe_flags WARNING: Logging before InitGoogleLogging() is written to STDERR W1010 20:58:34.316886 1749 flags.cc:294] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1010 20:58:34.316956 1749 flags.cc:294] Enabled unsafe flag: --never_fsync=true W1010 20:58:34.317415 1749 flags.cc:294] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.150.0 I1010 20:58:34.317509 1749 mem_tracker.cc:140] MemTracker: hard memory limit is 11.777661 GB I1010 20:58:34.317520 1749 mem_tracker.cc:142] MemTracker: soft memory limit is 7.066597 GB I1010 20:58:34.318480 1749 tablet_server_main.cc:54] Initializing tablet server... I1010 20:58:34.318626 1749 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 968115 I1010 20:58:34.319367 1749 server_base.cc:167] Could not load existing FS layout: Not found: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-0/instance: No such file or directory (error 2) I1010 20:58:34.319380 1749 server_base.cc:168] Creating new FS layout I1010 20:58:34.320413 1749 fs_manager.cc:354] Generated new instance metadata in path /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-0/instance: uuid: "d09b95f9213649f6ac7f29f92d8e3f11" format_stamp: "Formatted at 2016-10-10 20:58:34 on dist-test-slave-dist-test-slave-ymsa" I1010 20:58:34.321082 1749 fs_manager.cc:243] Opened local filesystem: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-0 uuid: "d09b95f9213649f6ac7f29f92d8e3f11" format_stamp: "Formatted at 2016-10-10 20:58:34 on dist-test-slave-dist-test-slave-ymsa" I1010 20:58:34.322780 1745 leader_election.cc:248] T 00000000000000000000000000000000 P d4cf8c406c3b4693b9535068d9267967 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I1010 20:58:34.323547 1749 tablet_server_main.cc:57] Starting tablet server... I1010 20:58:34.323556 1745 sys_catalog.cc:257] T 00000000000000000000000000000000 P d4cf8c406c3b4693b9535068d9267967 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletPeer. Latest consensus state: current_term: 1 config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4cf8c406c3b4693b9535068d9267967" member_type: VOTER } } I1010 20:58:34.323566 1745 sys_catalog.cc:260] T 00000000000000000000000000000000 P d4cf8c406c3b4693b9535068d9267967 [sys.catalog]: This master's current role is: FOLLOWER I1010 20:58:34.323580 1759 raft_consensus.cc:1981] T 00000000000000000000000000000000 P d4cf8c406c3b4693b9535068d9267967 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.602s I1010 20:58:34.323604 1745 sys_catalog.cc:335] T 00000000000000000000000000000000 P d4cf8c406c3b4693b9535068d9267967 [sys.catalog]: configured and running, proceeding with master startup. I1010 20:58:34.323612 1759 raft_consensus.cc:1861] T 00000000000000000000000000000000 P d4cf8c406c3b4693b9535068d9267967 [term 1 FOLLOWER]: Leader election won for term 1 W1010 20:58:34.323711 1762 catalog_manager.cc:365] Catalog manager background task thread going to sleep: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:58:34.323727 1691 master_main.cc:64] Master server successfully started. I1010 20:58:34.324141 1759 raft_consensus.cc:467] T 00000000000000000000000000000000 P d4cf8c406c3b4693b9535068d9267967 [term 1 LEADER]: Becoming Leader. State: Replica: d4cf8c406c3b4693b9535068d9267967, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:34.324162 1759 consensus_queue.cc:151] T 00000000000000000000000000000000 P d4cf8c406c3b4693b9535068d9267967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 1, Majority size: 1, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4cf8c406c3b4693b9535068d9267967" member_type: VOTER } I1010 20:58:34.325377 1760 sys_catalog.cc:257] T 00000000000000000000000000000000 P d4cf8c406c3b4693b9535068d9267967 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4cf8c406c3b4693b9535068d9267967" config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4cf8c406c3b4693b9535068d9267967" member_type: VOTER } } I1010 20:58:34.325389 1760 sys_catalog.cc:260] T 00000000000000000000000000000000 P d4cf8c406c3b4693b9535068d9267967 [sys.catalog]: This master's current role is: LEADER I1010 20:58:34.325392 1763 sys_catalog.cc:257] T 00000000000000000000000000000000 P d4cf8c406c3b4693b9535068d9267967 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4cf8c406c3b4693b9535068d9267967. Latest consensus state: current_term: 1 leader_uuid: "d4cf8c406c3b4693b9535068d9267967" config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4cf8c406c3b4693b9535068d9267967" member_type: VOTER } } I1010 20:58:34.325398 1763 sys_catalog.cc:260] T 00000000000000000000000000000000 P d4cf8c406c3b4693b9535068d9267967 [sys.catalog]: This master's current role is: LEADER I1010 20:58:34.325666 1781 catalog_manager.cc:660] Loading table and tablet metadata into memory... I1010 20:58:34.414142 1749 rpc_server.cc:164] RPC server started. Bound to: 127.6.150.0:51431 I1010 20:58:34.414212 1749 webserver.cc:125] Starting webserver on 127.6.150.0:0 I1010 20:58:34.414242 1749 webserver.cc:136] Document root disabled I1010 20:58:34.414364 1749 webserver.cc:220] Webserver started. Bound to: http://127.6.150.0:55339/ I1010 20:58:34.414489 1749 server_base.cc:239] Dumped server information to /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-0/info.pb I1010 20:58:34.414708 1749 tablet_server_main.cc:60] Tablet server successfully started. I1010 20:58:34.415859 1870 heartbeater.cc:292] Connected to a master server at 127.0.0.1:46501 I1010 20:58:34.415875 1870 heartbeater.cc:359] Registering TS with master... I1010 20:58:34.415884 1870 heartbeater.cc:373] Master 127.0.0.1:46501 requested a full tablet report, sending... I1010 20:58:34.416008 1712 ts_manager.cc:76] Registered new tserver with Master: d09b95f9213649f6ac7f29f92d8e3f11 (127.6.150.0:51431) I1010 20:58:34.423637 1686 external_mini_cluster.cc:622] Started /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver as pid 1749 I1010 20:58:34.423745 1686 external_mini_cluster.cc:586] Running /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-1 --fs_data_dirs=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-1 --rpc_bind_addresses=127.6.150.1:0 --local_ip_for_outbound_sockets=127.6.150.1 --webserver_interface=127.6.150.1 --webserver_port=0 --tserver_master_addrs=127.0.0.1:46501 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-1 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-1/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 --unlock_experimental_flags --unlock_unsafe_flags WARNING: Logging before InitGoogleLogging() is written to STDERR W1010 20:58:34.428082 1874 flags.cc:294] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1010 20:58:34.428155 1874 flags.cc:294] Enabled unsafe flag: --never_fsync=true W1010 20:58:34.428593 1874 flags.cc:294] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.150.1 I1010 20:58:34.428706 1874 mem_tracker.cc:140] MemTracker: hard memory limit is 11.777661 GB I1010 20:58:34.428714 1874 mem_tracker.cc:142] MemTracker: soft memory limit is 7.066597 GB I1010 20:58:34.429685 1874 tablet_server_main.cc:54] Initializing tablet server... I1010 20:58:34.429811 1874 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 968115 I1010 20:58:34.430443 1874 server_base.cc:167] Could not load existing FS layout: Not found: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-1/instance: No such file or directory (error 2) I1010 20:58:34.430452 1874 server_base.cc:168] Creating new FS layout I1010 20:58:34.431577 1874 fs_manager.cc:354] Generated new instance metadata in path /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-1/instance: uuid: "7160a237fa2e437da8305b75ed2e6ec8" format_stamp: "Formatted at 2016-10-10 20:58:34 on dist-test-slave-dist-test-slave-ymsa" I1010 20:58:34.432162 1874 fs_manager.cc:243] Opened local filesystem: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-1 uuid: "7160a237fa2e437da8305b75ed2e6ec8" format_stamp: "Formatted at 2016-10-10 20:58:34 on dist-test-slave-dist-test-slave-ymsa" I1010 20:58:34.434703 1874 tablet_server_main.cc:57] Starting tablet server... I1010 20:58:34.444339 1874 rpc_server.cc:164] RPC server started. Bound to: 127.6.150.1:57815 I1010 20:58:34.444388 1874 webserver.cc:125] Starting webserver on 127.6.150.1:0 I1010 20:58:34.444398 1874 webserver.cc:136] Document root disabled I1010 20:58:34.444553 1874 webserver.cc:220] Webserver started. Bound to: http://127.6.150.1:37972/ I1010 20:58:34.444669 1874 server_base.cc:239] Dumped server information to /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-1/info.pb I1010 20:58:34.444942 1874 tablet_server_main.cc:60] Tablet server successfully started. I1010 20:58:34.445998 1988 heartbeater.cc:292] Connected to a master server at 127.0.0.1:46501 I1010 20:58:34.446017 1988 heartbeater.cc:359] Registering TS with master... I1010 20:58:34.446030 1988 heartbeater.cc:373] Master 127.0.0.1:46501 requested a full tablet report, sending... I1010 20:58:34.446125 1712 ts_manager.cc:76] Registered new tserver with Master: 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815) I1010 20:58:34.454457 1686 external_mini_cluster.cc:622] Started /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver as pid 1874 I1010 20:58:34.454536 1686 external_mini_cluster.cc:586] Running /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-2 --fs_data_dirs=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-2 --rpc_bind_addresses=127.6.150.2:0 --local_ip_for_outbound_sockets=127.6.150.2 --webserver_interface=127.6.150.2 --webserver_port=0 --tserver_master_addrs=127.0.0.1:46501 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-2 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-2/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 --unlock_experimental_flags --unlock_unsafe_flags WARNING: Logging before InitGoogleLogging() is written to STDERR W1010 20:58:34.459756 1991 flags.cc:294] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1010 20:58:34.459821 1991 flags.cc:294] Enabled unsafe flag: --never_fsync=true W1010 20:58:34.460327 1991 flags.cc:294] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.150.2 I1010 20:58:34.460444 1991 mem_tracker.cc:140] MemTracker: hard memory limit is 11.777661 GB I1010 20:58:34.460458 1991 mem_tracker.cc:142] MemTracker: soft memory limit is 7.066597 GB I1010 20:58:34.461577 1991 tablet_server_main.cc:54] Initializing tablet server... I1010 20:58:34.461730 1991 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 968115 I1010 20:58:34.462570 1991 server_base.cc:167] Could not load existing FS layout: Not found: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-2/instance: No such file or directory (error 2) I1010 20:58:34.462582 1991 server_base.cc:168] Creating new FS layout I1010 20:58:34.463738 1991 fs_manager.cc:354] Generated new instance metadata in path /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-2/instance: uuid: "75e56306e836413685f502a571284009" format_stamp: "Formatted at 2016-10-10 20:58:34 on dist-test-slave-dist-test-slave-ymsa" I1010 20:58:34.464395 1991 fs_manager.cc:243] Opened local filesystem: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-2 uuid: "75e56306e836413685f502a571284009" format_stamp: "Formatted at 2016-10-10 20:58:34 on dist-test-slave-dist-test-slave-ymsa" I1010 20:58:34.467049 1991 tablet_server_main.cc:57] Starting tablet server... I1010 20:58:34.475998 1991 rpc_server.cc:164] RPC server started. Bound to: 127.6.150.2:38018 I1010 20:58:34.476050 1991 webserver.cc:125] Starting webserver on 127.6.150.2:0 I1010 20:58:34.476068 1991 webserver.cc:136] Document root disabled I1010 20:58:34.476167 1991 webserver.cc:220] Webserver started. Bound to: http://127.6.150.2:36092/ I1010 20:58:34.476272 1991 server_base.cc:239] Dumped server information to /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-2/info.pb I1010 20:58:34.476444 1991 tablet_server_main.cc:60] Tablet server successfully started. I1010 20:58:34.477280 2105 heartbeater.cc:292] Connected to a master server at 127.0.0.1:46501 I1010 20:58:34.477296 2105 heartbeater.cc:359] Registering TS with master... I1010 20:58:34.477306 2105 heartbeater.cc:373] Master 127.0.0.1:46501 requested a full tablet report, sending... I1010 20:58:34.477459 1712 ts_manager.cc:76] Registered new tserver with Master: 75e56306e836413685f502a571284009 (127.6.150.2:38018) I1010 20:58:34.485374 1686 external_mini_cluster.cc:622] Started /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver as pid 1991 I1010 20:58:34.486888 1686 external_mini_cluster.cc:341] 3 TS(s) registered with all masters I1010 20:58:34.489796 1686 external_mini_cluster.cc:687] Killing /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver with pid 1874 I1010 20:58:34.492161 1686 external_mini_cluster.cc:687] Killing /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver with pid 1991 I1010 20:58:34.494557 1712 catalog_manager.cc:789] CreateTable from {real_user=slave, eff_user=} at 127.0.0.1:46414: name: "test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W1010 20:58:34.495874 1696 catalog_manager.cc:2301] TS 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815): Create Tablet RPC failed for tablet b1e6a7b25cce46d9a64325794390ab21: Network error: Client connection negotiation failed: client connection to 127.6.150.1:57815: connect: Connection refused (error 111) I1010 20:58:34.495896 1696 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet b1e6a7b25cce46d9a64325794390ab21 on TS 7160a237fa2e437da8305b75ed2e6ec8 with a delay of 60ms (attempt = 1)... W1010 20:58:34.495964 1695 catalog_manager.cc:2301] TS 75e56306e836413685f502a571284009 (127.6.150.2:38018): Create Tablet RPC failed for tablet b1e6a7b25cce46d9a64325794390ab21: Network error: Client connection negotiation failed: client connection to 127.6.150.2:38018: connect: Connection refused (error 111) I1010 20:58:34.495980 1695 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet b1e6a7b25cce46d9a64325794390ab21 on TS 75e56306e836413685f502a571284009 with a delay of 35ms (attempt = 1)... I1010 20:58:34.496637 1807 tablet_service.cc:619] Processing CreateTablet for tablet b1e6a7b25cce46d9a64325794390ab21 (table=test-table [id=cdc4846c3fe84fb7b1522be60094eca9]), partition=range: [, ) I1010 20:58:34.497520 1807 ts_tablet_manager.cc:766] Registered tablet b1e6a7b25cce46d9a64325794390ab21 I1010 20:58:34.497689 2116 ts_tablet_manager.cc:632] T b1e6a7b25cce46d9a64325794390ab21 P d09b95f9213649f6ac7f29f92d8e3f11: Bootstrapping tablet I1010 20:58:34.497865 2116 tablet_bootstrap.cc:380] T b1e6a7b25cce46d9a64325794390ab21 P d09b95f9213649f6ac7f29f92d8e3f11: Bootstrap starting. I1010 20:58:34.498054 2116 tablet_bootstrap.cc:542] T b1e6a7b25cce46d9a64325794390ab21 P d09b95f9213649f6ac7f29f92d8e3f11: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:34.498183 2116 tablet_bootstrap.cc:482] T b1e6a7b25cce46d9a64325794390ab21 P d09b95f9213649f6ac7f29f92d8e3f11: No blocks or log segments found. Creating new log. I1010 20:58:34.498343 2116 log.cc:347] Log is configured to *not* fsync() on all Append() calls I1010 20:58:34.498829 2116 tablet_bootstrap.cc:380] T b1e6a7b25cce46d9a64325794390ab21 P d09b95f9213649f6ac7f29f92d8e3f11: No bootstrap required, opened a new log I1010 20:58:34.498853 2116 ts_tablet_manager.cc:637] T b1e6a7b25cce46d9a64325794390ab21 P d09b95f9213649f6ac7f29f92d8e3f11: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:34.499231 2116 raft_consensus.cc:275] T b1e6a7b25cce46d9a64325794390ab21 P d09b95f9213649f6ac7f29f92d8e3f11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7160a237fa2e437da8305b75ed2e6ec8" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 57815 } } peers { permanent_uuid: "75e56306e836413685f502a571284009" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 38018 } } peers { permanent_uuid: "d09b95f9213649f6ac7f29f92d8e3f11" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 51431 } } I1010 20:58:34.499251 2116 raft_consensus.cc:306] T b1e6a7b25cce46d9a64325794390ab21 P d09b95f9213649f6ac7f29f92d8e3f11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:34.499260 2116 raft_consensus.cc:502] T b1e6a7b25cce46d9a64325794390ab21 P d09b95f9213649f6ac7f29f92d8e3f11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09b95f9213649f6ac7f29f92d8e3f11, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:34.499270 2116 consensus_queue.cc:168] T b1e6a7b25cce46d9a64325794390ab21 P d09b95f9213649f6ac7f29f92d8e3f11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:34.499416 2116 ts_tablet_manager.cc:664] T b1e6a7b25cce46d9a64325794390ab21 P d09b95f9213649f6ac7f29f92d8e3f11: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:34.499426 1870 heartbeater.cc:365] Master 127.0.0.1:46501 was elected leader, sending a full tablet report... W1010 20:58:34.531499 1695 catalog_manager.cc:2301] TS 75e56306e836413685f502a571284009 (127.6.150.2:38018): Create Tablet RPC failed for tablet b1e6a7b25cce46d9a64325794390ab21: Network error: Client connection negotiation failed: client connection to 127.6.150.2:38018: connect: Connection refused (error 111) I1010 20:58:34.531524 1695 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet b1e6a7b25cce46d9a64325794390ab21 on TS 75e56306e836413685f502a571284009 with a delay of 70ms (attempt = 2)... W1010 20:58:34.556161 1696 catalog_manager.cc:2301] TS 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815): Create Tablet RPC failed for tablet b1e6a7b25cce46d9a64325794390ab21: Network error: Client connection negotiation failed: client connection to 127.6.150.1:57815: connect: Connection refused (error 111) I1010 20:58:34.556176 1696 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet b1e6a7b25cce46d9a64325794390ab21 on TS 7160a237fa2e437da8305b75ed2e6ec8 with a delay of 33ms (attempt = 2)... W1010 20:58:34.589473 1696 catalog_manager.cc:2301] TS 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815): Create Tablet RPC failed for tablet b1e6a7b25cce46d9a64325794390ab21: Network error: Client connection negotiation failed: client connection to 127.6.150.1:57815: connect: Connection refused (error 111) I1010 20:58:34.589489 1696 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet b1e6a7b25cce46d9a64325794390ab21 on TS 7160a237fa2e437da8305b75ed2e6ec8 with a delay of 90ms (attempt = 3)... I1010 20:58:34.596321 1686 create-table-itest.cc:83] Waiting for the master to retry creating the tablet 3 times... 1 RPCs seen so far W1010 20:58:34.602066 1695 catalog_manager.cc:2301] TS 75e56306e836413685f502a571284009 (127.6.150.2:38018): Create Tablet RPC failed for tablet b1e6a7b25cce46d9a64325794390ab21: Network error: Client connection negotiation failed: client connection to 127.6.150.2:38018: connect: Connection refused (error 111) I1010 20:58:34.602082 1695 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet b1e6a7b25cce46d9a64325794390ab21 on TS 75e56306e836413685f502a571284009 with a delay of 97ms (attempt = 3)... W1010 20:58:34.680122 1696 catalog_manager.cc:2301] TS 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815): Create Tablet RPC failed for tablet b1e6a7b25cce46d9a64325794390ab21: Network error: Client connection negotiation failed: client connection to 127.6.150.1:57815: connect: Connection refused (error 111) I1010 20:58:34.680182 1696 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet b1e6a7b25cce46d9a64325794390ab21 on TS 7160a237fa2e437da8305b75ed2e6ec8 with a delay of 147ms (attempt = 4)... I1010 20:58:34.697093 1686 create-table-itest.cc:83] Waiting for the master to retry creating the tablet 3 times... 1 RPCs seen so far W1010 20:58:34.699456 1695 catalog_manager.cc:2301] TS 75e56306e836413685f502a571284009 (127.6.150.2:38018): Create Tablet RPC failed for tablet b1e6a7b25cce46d9a64325794390ab21: Network error: Client connection negotiation failed: client connection to 127.6.150.2:38018: connect: Connection refused (error 111) I1010 20:58:34.699471 1695 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet b1e6a7b25cce46d9a64325794390ab21 on TS 75e56306e836413685f502a571284009 with a delay of 161ms (attempt = 4)... I1010 20:58:34.797880 1686 create-table-itest.cc:83] Waiting for the master to retry creating the tablet 3 times... 1 RPCs seen so far W1010 20:58:34.827528 1696 catalog_manager.cc:2301] TS 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815): Create Tablet RPC failed for tablet b1e6a7b25cce46d9a64325794390ab21: Network error: Client connection negotiation failed: client connection to 127.6.150.1:57815: connect: Connection refused (error 111) I1010 20:58:34.827550 1696 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet b1e6a7b25cce46d9a64325794390ab21 on TS 7160a237fa2e437da8305b75ed2e6ec8 with a delay of 299ms (attempt = 5)... W1010 20:58:34.861059 1695 catalog_manager.cc:2301] TS 75e56306e836413685f502a571284009 (127.6.150.2:38018): Create Tablet RPC failed for tablet b1e6a7b25cce46d9a64325794390ab21: Network error: Client connection negotiation failed: client connection to 127.6.150.2:38018: connect: Connection refused (error 111) I1010 20:58:34.861083 1695 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet b1e6a7b25cce46d9a64325794390ab21 on TS 75e56306e836413685f502a571284009 with a delay of 260ms (attempt = 5)... I1010 20:58:34.898669 1686 create-table-itest.cc:83] Waiting for the master to retry creating the tablet 3 times... 1 RPCs seen so far I1010 20:58:34.906276 2119 raft_consensus.cc:375] T b1e6a7b25cce46d9a64325794390ab21 P d09b95f9213649f6ac7f29f92d8e3f11 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:34.906297 2119 raft_consensus.cc:2031] T b1e6a7b25cce46d9a64325794390ab21 P d09b95f9213649f6ac7f29f92d8e3f11 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:34.906622 2119 raft_consensus.cc:1981] T b1e6a7b25cce46d9a64325794390ab21 P d09b95f9213649f6ac7f29f92d8e3f11 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.614s I1010 20:58:34.906690 2119 raft_consensus.cc:391] T b1e6a7b25cce46d9a64325794390ab21 P d09b95f9213649f6ac7f29f92d8e3f11 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7160a237fa2e437da8305b75ed2e6ec8" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 57815 } } peers { permanent_uuid: "75e56306e836413685f502a571284009" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 38018 } } peers { permanent_uuid: "d09b95f9213649f6ac7f29f92d8e3f11" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 51431 } } I1010 20:58:34.999398 1686 create-table-itest.cc:83] Waiting for the master to retry creating the tablet 3 times... 1 RPCs seen so far I1010 20:58:35.055634 2119 leader_election.cc:223] T b1e6a7b25cce46d9a64325794390ab21 P d09b95f9213649f6ac7f29f92d8e3f11 [CANDIDATE]: Term 1 election: Requesting vote from peer 7160a237fa2e437da8305b75ed2e6ec8 I1010 20:58:35.055680 2119 leader_election.cc:223] T b1e6a7b25cce46d9a64325794390ab21 P d09b95f9213649f6ac7f29f92d8e3f11 [CANDIDATE]: Term 1 election: Requesting vote from peer 75e56306e836413685f502a571284009 W1010 20:58:35.056236 1754 leader_election.cc:275] T b1e6a7b25cce46d9a64325794390ab21 P d09b95f9213649f6ac7f29f92d8e3f11 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 7160a237fa2e437da8305b75ed2e6ec8: Network error: Client connection negotiation failed: client connection to 127.6.150.1:57815: connect: Connection refused (error 111) W1010 20:58:35.056287 1753 leader_election.cc:275] T b1e6a7b25cce46d9a64325794390ab21 P d09b95f9213649f6ac7f29f92d8e3f11 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 75e56306e836413685f502a571284009: Network error: Client connection negotiation failed: client connection to 127.6.150.2:38018: connect: Connection refused (error 111) I1010 20:58:35.056318 1753 leader_election.cc:248] T b1e6a7b25cce46d9a64325794390ab21 P d09b95f9213649f6ac7f29f92d8e3f11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:35.056473 2124 raft_consensus.cc:1981] T b1e6a7b25cce46d9a64325794390ab21 P d09b95f9213649f6ac7f29f92d8e3f11 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.525s I1010 20:58:35.056491 2124 raft_consensus.cc:1823] T b1e6a7b25cce46d9a64325794390ab21 P d09b95f9213649f6ac7f29f92d8e3f11 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given I1010 20:58:35.100250 1686 create-table-itest.cc:83] Waiting for the master to retry creating the tablet 3 times... 1 RPCs seen so far W1010 20:58:35.122014 1695 catalog_manager.cc:2301] TS 75e56306e836413685f502a571284009 (127.6.150.2:38018): Create Tablet RPC failed for tablet b1e6a7b25cce46d9a64325794390ab21: Network error: Client connection negotiation failed: client connection to 127.6.150.2:38018: connect: Connection refused (error 111) I1010 20:58:35.122041 1695 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet b1e6a7b25cce46d9a64325794390ab21 on TS 75e56306e836413685f502a571284009 with a delay of 363ms (attempt = 6)... W1010 20:58:35.126839 1696 catalog_manager.cc:2301] TS 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815): Create Tablet RPC failed for tablet b1e6a7b25cce46d9a64325794390ab21: Network error: Client connection negotiation failed: client connection to 127.6.150.1:57815: connect: Connection refused (error 111) I1010 20:58:35.126858 1696 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet b1e6a7b25cce46d9a64325794390ab21 on TS 7160a237fa2e437da8305b75ed2e6ec8 with a delay of 358ms (attempt = 6)... I1010 20:58:35.201036 1686 create-table-itest.cc:83] Waiting for the master to retry creating the tablet 3 times... 1 RPCs seen so far I1010 20:58:35.301945 1686 create-table-itest.cc:83] Waiting for the master to retry creating the tablet 3 times... 1 RPCs seen so far I1010 20:58:35.403030 1686 create-table-itest.cc:83] Waiting for the master to retry creating the tablet 3 times... 1 RPCs seen so far W1010 20:58:35.485818 1696 catalog_manager.cc:2301] TS 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815): Create Tablet RPC failed for tablet b1e6a7b25cce46d9a64325794390ab21: Network error: Client connection negotiation failed: client connection to 127.6.150.1:57815: connect: Connection refused (error 111) W1010 20:58:35.485852 1696 catalog_manager.cc:2352] Request timed out: CreateTablet RPC for tablet b1e6a7b25cce46d9a64325794390ab21 on TS 7160a237fa2e437da8305b75ed2e6ec8 W1010 20:58:35.486068 1695 catalog_manager.cc:2301] TS 75e56306e836413685f502a571284009 (127.6.150.2:38018): Create Tablet RPC failed for tablet b1e6a7b25cce46d9a64325794390ab21: Network error: Client connection negotiation failed: client connection to 127.6.150.2:38018: connect: Connection refused (error 111) W1010 20:58:35.486099 1695 catalog_manager.cc:2352] Request timed out: CreateTablet RPC for tablet b1e6a7b25cce46d9a64325794390ab21 on TS 75e56306e836413685f502a571284009 W1010 20:58:35.496055 1762 catalog_manager.cc:2975] Tablet b1e6a7b25cce46d9a64325794390ab21 (table test-table [id=cdc4846c3fe84fb7b1522be60094eca9]) was not created within the allowed timeout. Replacing with a new tablet 6e8c16ee547a46c2ae96a84f02357575 I1010 20:58:35.496831 1762 catalog_manager.cc:2864] Sending DeleteTablet for 3 replicas of tablet b1e6a7b25cce46d9a64325794390ab21 I1010 20:58:35.496856 1762 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b1e6a7b25cce46d9a64325794390ab21 on 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815) (Replaced by 6e8c16ee547a46c2ae96a84f02357575 at 2016-10-10 20:58:35 UTC) I1010 20:58:35.496875 1762 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b1e6a7b25cce46d9a64325794390ab21 on 75e56306e836413685f502a571284009 (127.6.150.2:38018) (Replaced by 6e8c16ee547a46c2ae96a84f02357575 at 2016-10-10 20:58:35 UTC) I1010 20:58:35.496886 1762 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b1e6a7b25cce46d9a64325794390ab21 on d09b95f9213649f6ac7f29f92d8e3f11 (127.6.150.0:51431) (Replaced by 6e8c16ee547a46c2ae96a84f02357575 at 2016-10-10 20:58:35 UTC) I1010 20:58:35.497114 1807 tablet_service.cc:660] Processing DeleteTablet for tablet b1e6a7b25cce46d9a64325794390ab21 with delete_type TABLET_DATA_DELETED (Replaced by 6e8c16ee547a46c2ae96a84f02357575 at 2016-10-10 20:58:35 UTC) from {real_user=slave, eff_user=} at 127.0.0.1:43898 W1010 20:58:35.497125 1695 catalog_manager.cc:2301] TS 75e56306e836413685f502a571284009 (127.6.150.2:38018): Create Tablet RPC failed for tablet 6e8c16ee547a46c2ae96a84f02357575: Network error: Client connection negotiation failed: client connection to 127.6.150.2:38018: connect: Connection refused (error 111) I1010 20:58:35.497136 1695 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 6e8c16ee547a46c2ae96a84f02357575 on TS 75e56306e836413685f502a571284009 with a delay of 27ms (attempt = 1)... I1010 20:58:35.497139 1807 tablet_peer.cc:215] Initiating TabletPeer shutdown for tablet: b1e6a7b25cce46d9a64325794390ab21 W1010 20:58:35.497145 1695 catalog_manager.cc:2301] TS 75e56306e836413685f502a571284009 (127.6.150.2:38018): Delete Tablet RPC failed for tablet b1e6a7b25cce46d9a64325794390ab21: Network error: Client connection negotiation failed: client connection to 127.6.150.2:38018: connect: Connection refused (error 111) I1010 20:58:35.497148 1807 maintenance_manager.cc:182] Unregistered op CompactRowSetsOp(b1e6a7b25cce46d9a64325794390ab21) I1010 20:58:35.497150 1695 catalog_manager.cc:2355] Scheduling retry of b1e6a7b25cce46d9a64325794390ab21 Delete Tablet RPC for TS=75e56306e836413685f502a571284009 with a delay of 33ms (attempt = 1)... I1010 20:58:35.497153 1807 maintenance_manager.cc:182] Unregistered op MinorDeltaCompactionOp(b1e6a7b25cce46d9a64325794390ab21) I1010 20:58:35.497149 1806 tablet_service.cc:619] Processing CreateTablet for tablet 6e8c16ee547a46c2ae96a84f02357575 (table=test-table [id=cdc4846c3fe84fb7b1522be60094eca9]), partition=range: [, ) W1010 20:58:35.497215 1696 catalog_manager.cc:2301] TS 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815): Create Tablet RPC failed for tablet 6e8c16ee547a46c2ae96a84f02357575: Network error: Client connection negotiation failed: client connection to 127.6.150.1:57815: connect: Connection refused (error 111) I1010 20:58:35.497228 1696 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 6e8c16ee547a46c2ae96a84f02357575 on TS 7160a237fa2e437da8305b75ed2e6ec8 with a delay of 28ms (attempt = 1)... W1010 20:58:35.497236 1696 catalog_manager.cc:2301] TS 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815): Delete Tablet RPC failed for tablet b1e6a7b25cce46d9a64325794390ab21: Network error: Client connection negotiation failed: client connection to 127.6.150.1:57815: connect: Connection refused (error 111) I1010 20:58:35.497239 1696 catalog_manager.cc:2355] Scheduling retry of b1e6a7b25cce46d9a64325794390ab21 Delete Tablet RPC for TS=7160a237fa2e437da8305b75ed2e6ec8 with a delay of 58ms (attempt = 1)... I1010 20:58:35.497619 1806 ts_tablet_manager.cc:766] Registered tablet 6e8c16ee547a46c2ae96a84f02357575 I1010 20:58:35.497159 1807 maintenance_manager.cc:182] Unregistered op MajorDeltaCompactionOp(b1e6a7b25cce46d9a64325794390ab21) I1010 20:58:35.497668 1807 maintenance_manager.cc:182] Unregistered op FlushMRSOp(b1e6a7b25cce46d9a64325794390ab21) I1010 20:58:35.497673 1807 maintenance_manager.cc:182] Unregistered op FlushDeltaMemStoresOp(b1e6a7b25cce46d9a64325794390ab21) I1010 20:58:35.497680 1807 maintenance_manager.cc:182] Unregistered op LogGCOp(b1e6a7b25cce46d9a64325794390ab21) I1010 20:58:35.497695 1807 raft_consensus.cc:1495] T b1e6a7b25cce46d9a64325794390ab21 P d09b95f9213649f6ac7f29f92d8e3f11 [term 1 FOLLOWER]: Raft consensus shutting down. I1010 20:58:35.497704 1807 raft_consensus.cc:1511] T b1e6a7b25cce46d9a64325794390ab21 P d09b95f9213649f6ac7f29f92d8e3f11 [term 1 FOLLOWER]: Raft consensus is shut down! I1010 20:58:35.497838 2128 ts_tablet_manager.cc:632] T 6e8c16ee547a46c2ae96a84f02357575 P d09b95f9213649f6ac7f29f92d8e3f11: Bootstrapping tablet I1010 20:58:35.497992 2128 tablet_bootstrap.cc:380] T 6e8c16ee547a46c2ae96a84f02357575 P d09b95f9213649f6ac7f29f92d8e3f11: Bootstrap starting. I1010 20:58:35.498019 1807 ts_tablet_manager.cc:916] T b1e6a7b25cce46d9a64325794390ab21 P d09b95f9213649f6ac7f29f92d8e3f11: Deleting tablet data with delete state TABLET_DATA_DELETED I1010 20:58:35.498095 2128 tablet_bootstrap.cc:542] T 6e8c16ee547a46c2ae96a84f02357575 P d09b95f9213649f6ac7f29f92d8e3f11: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:35.498231 2128 tablet_bootstrap.cc:482] T 6e8c16ee547a46c2ae96a84f02357575 P d09b95f9213649f6ac7f29f92d8e3f11: No blocks or log segments found. Creating new log. I1010 20:58:35.498759 2128 tablet_bootstrap.cc:380] T 6e8c16ee547a46c2ae96a84f02357575 P d09b95f9213649f6ac7f29f92d8e3f11: No bootstrap required, opened a new log I1010 20:58:35.498886 2128 ts_tablet_manager.cc:637] T 6e8c16ee547a46c2ae96a84f02357575 P d09b95f9213649f6ac7f29f92d8e3f11: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:35.499228 2128 raft_consensus.cc:275] T 6e8c16ee547a46c2ae96a84f02357575 P d09b95f9213649f6ac7f29f92d8e3f11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e56306e836413685f502a571284009" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 38018 } } peers { permanent_uuid: "7160a237fa2e437da8305b75ed2e6ec8" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 57815 } } peers { permanent_uuid: "d09b95f9213649f6ac7f29f92d8e3f11" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 51431 } } I1010 20:58:35.499243 2128 raft_consensus.cc:306] T 6e8c16ee547a46c2ae96a84f02357575 P d09b95f9213649f6ac7f29f92d8e3f11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:35.499263 2128 raft_consensus.cc:502] T 6e8c16ee547a46c2ae96a84f02357575 P d09b95f9213649f6ac7f29f92d8e3f11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09b95f9213649f6ac7f29f92d8e3f11, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:35.499281 2128 consensus_queue.cc:168] T 6e8c16ee547a46c2ae96a84f02357575 P d09b95f9213649f6ac7f29f92d8e3f11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:35.499445 2128 ts_tablet_manager.cc:664] T 6e8c16ee547a46c2ae96a84f02357575 P d09b95f9213649f6ac7f29f92d8e3f11: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:35.503948 1686 create-table-itest.cc:83] Waiting for the master to retry creating the tablet 3 times... 2 RPCs seen so far W1010 20:58:35.524653 1695 catalog_manager.cc:2301] TS 75e56306e836413685f502a571284009 (127.6.150.2:38018): Create Tablet RPC failed for tablet 6e8c16ee547a46c2ae96a84f02357575: Network error: Client connection negotiation failed: client connection to 127.6.150.2:38018: connect: Connection refused (error 111) I1010 20:58:35.524669 1695 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 6e8c16ee547a46c2ae96a84f02357575 on TS 75e56306e836413685f502a571284009 with a delay of 61ms (attempt = 2)... W1010 20:58:35.526463 1696 catalog_manager.cc:2301] TS 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815): Create Tablet RPC failed for tablet 6e8c16ee547a46c2ae96a84f02357575: Network error: Client connection negotiation failed: client connection to 127.6.150.1:57815: connect: Connection refused (error 111) I1010 20:58:35.526476 1696 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 6e8c16ee547a46c2ae96a84f02357575 on TS 7160a237fa2e437da8305b75ed2e6ec8 with a delay of 34ms (attempt = 2)... I1010 20:58:35.530872 1695 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b1e6a7b25cce46d9a64325794390ab21 on 75e56306e836413685f502a571284009 (127.6.150.2:38018) (Replaced by 6e8c16ee547a46c2ae96a84f02357575 at 2016-10-10 20:58:35 UTC) W1010 20:58:35.531051 1695 catalog_manager.cc:2301] TS 75e56306e836413685f502a571284009 (127.6.150.2:38018): Delete Tablet RPC failed for tablet b1e6a7b25cce46d9a64325794390ab21: Network error: Client connection negotiation failed: client connection to 127.6.150.2:38018: connect: Connection refused (error 111) I1010 20:58:35.531075 1695 catalog_manager.cc:2355] Scheduling retry of b1e6a7b25cce46d9a64325794390ab21 Delete Tablet RPC for TS=75e56306e836413685f502a571284009 with a delay of 50ms (attempt = 2)... I1010 20:58:35.555661 1696 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b1e6a7b25cce46d9a64325794390ab21 on 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815) (Replaced by 6e8c16ee547a46c2ae96a84f02357575 at 2016-10-10 20:58:35 UTC) W1010 20:58:35.555791 1696 catalog_manager.cc:2301] TS 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815): Delete Tablet RPC failed for tablet b1e6a7b25cce46d9a64325794390ab21: Network error: Client connection negotiation failed: client connection to 127.6.150.1:57815: connect: Connection refused (error 111) I1010 20:58:35.555804 1696 catalog_manager.cc:2355] Scheduling retry of b1e6a7b25cce46d9a64325794390ab21 Delete Tablet RPC for TS=7160a237fa2e437da8305b75ed2e6ec8 with a delay of 63ms (attempt = 2)... W1010 20:58:35.561095 1696 catalog_manager.cc:2301] TS 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815): Create Tablet RPC failed for tablet 6e8c16ee547a46c2ae96a84f02357575: Network error: Client connection negotiation failed: client connection to 127.6.150.1:57815: connect: Connection refused (error 111) I1010 20:58:35.561107 1696 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 6e8c16ee547a46c2ae96a84f02357575 on TS 7160a237fa2e437da8305b75ed2e6ec8 with a delay of 66ms (attempt = 3)... I1010 20:58:35.572461 1807 ts_tablet_manager.cc:926] T b1e6a7b25cce46d9a64325794390ab21 P d09b95f9213649f6ac7f29f92d8e3f11: Tablet deleted. Last logged OpId: 0.0 I1010 20:58:35.572937 1697 catalog_manager.cc:2567] TS d09b95f9213649f6ac7f29f92d8e3f11 (127.6.150.0:51431): tablet b1e6a7b25cce46d9a64325794390ab21 (table test-table [id=cdc4846c3fe84fb7b1522be60094eca9]) successfully deleted I1010 20:58:35.581266 1695 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b1e6a7b25cce46d9a64325794390ab21 on 75e56306e836413685f502a571284009 (127.6.150.2:38018) (Replaced by 6e8c16ee547a46c2ae96a84f02357575 at 2016-10-10 20:58:35 UTC) W1010 20:58:35.581403 1695 catalog_manager.cc:2301] TS 75e56306e836413685f502a571284009 (127.6.150.2:38018): Delete Tablet RPC failed for tablet b1e6a7b25cce46d9a64325794390ab21: Network error: Client connection negotiation failed: client connection to 127.6.150.2:38018: connect: Connection refused (error 111) I1010 20:58:35.581424 1695 catalog_manager.cc:2355] Scheduling retry of b1e6a7b25cce46d9a64325794390ab21 Delete Tablet RPC for TS=75e56306e836413685f502a571284009 with a delay of 88ms (attempt = 3)... W1010 20:58:35.586814 1695 catalog_manager.cc:2301] TS 75e56306e836413685f502a571284009 (127.6.150.2:38018): Create Tablet RPC failed for tablet 6e8c16ee547a46c2ae96a84f02357575: Network error: Client connection negotiation failed: client connection to 127.6.150.2:38018: connect: Connection refused (error 111) I1010 20:58:35.586830 1695 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 6e8c16ee547a46c2ae96a84f02357575 on TS 75e56306e836413685f502a571284009 with a delay of 71ms (attempt = 3)... I1010 20:58:35.604681 1686 create-table-itest.cc:83] Waiting for the master to retry creating the tablet 3 times... 2 RPCs seen so far I1010 20:58:35.619079 1696 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b1e6a7b25cce46d9a64325794390ab21 on 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815) (Replaced by 6e8c16ee547a46c2ae96a84f02357575 at 2016-10-10 20:58:35 UTC) W1010 20:58:35.619261 1696 catalog_manager.cc:2301] TS 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815): Delete Tablet RPC failed for tablet b1e6a7b25cce46d9a64325794390ab21: Network error: Client connection negotiation failed: client connection to 127.6.150.1:57815: connect: Connection refused (error 111) I1010 20:58:35.619271 1696 catalog_manager.cc:2355] Scheduling retry of b1e6a7b25cce46d9a64325794390ab21 Delete Tablet RPC for TS=7160a237fa2e437da8305b75ed2e6ec8 with a delay of 82ms (attempt = 3)... W1010 20:58:35.627645 1696 catalog_manager.cc:2301] TS 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815): Create Tablet RPC failed for tablet 6e8c16ee547a46c2ae96a84f02357575: Network error: Client connection negotiation failed: client connection to 127.6.150.1:57815: connect: Connection refused (error 111) I1010 20:58:35.627662 1696 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 6e8c16ee547a46c2ae96a84f02357575 on TS 7160a237fa2e437da8305b75ed2e6ec8 with a delay of 144ms (attempt = 4)... W1010 20:58:35.658319 1695 catalog_manager.cc:2301] TS 75e56306e836413685f502a571284009 (127.6.150.2:38018): Create Tablet RPC failed for tablet 6e8c16ee547a46c2ae96a84f02357575: Network error: Client connection negotiation failed: client connection to 127.6.150.2:38018: connect: Connection refused (error 111) I1010 20:58:35.658334 1695 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 6e8c16ee547a46c2ae96a84f02357575 on TS 75e56306e836413685f502a571284009 with a delay of 161ms (attempt = 4)... I1010 20:58:35.670541 1695 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b1e6a7b25cce46d9a64325794390ab21 on 75e56306e836413685f502a571284009 (127.6.150.2:38018) (Replaced by 6e8c16ee547a46c2ae96a84f02357575 at 2016-10-10 20:58:35 UTC) W1010 20:58:35.670671 1695 catalog_manager.cc:2301] TS 75e56306e836413685f502a571284009 (127.6.150.2:38018): Delete Tablet RPC failed for tablet b1e6a7b25cce46d9a64325794390ab21: Network error: Client connection negotiation failed: client connection to 127.6.150.2:38018: connect: Connection refused (error 111) I1010 20:58:35.670691 1695 catalog_manager.cc:2355] Scheduling retry of b1e6a7b25cce46d9a64325794390ab21 Delete Tablet RPC for TS=75e56306e836413685f502a571284009 with a delay of 155ms (attempt = 4)... I1010 20:58:35.701949 1696 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b1e6a7b25cce46d9a64325794390ab21 on 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815) (Replaced by 6e8c16ee547a46c2ae96a84f02357575 at 2016-10-10 20:58:35 UTC) W1010 20:58:35.702085 1696 catalog_manager.cc:2301] TS 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815): Delete Tablet RPC failed for tablet b1e6a7b25cce46d9a64325794390ab21: Network error: Client connection negotiation failed: client connection to 127.6.150.1:57815: connect: Connection refused (error 111) I1010 20:58:35.702097 1696 catalog_manager.cc:2355] Scheduling retry of b1e6a7b25cce46d9a64325794390ab21 Delete Tablet RPC for TS=7160a237fa2e437da8305b75ed2e6ec8 with a delay of 165ms (attempt = 4)... I1010 20:58:35.705468 1686 create-table-itest.cc:83] Waiting for the master to retry creating the tablet 3 times... 2 RPCs seen so far W1010 20:58:35.772006 1696 catalog_manager.cc:2301] TS 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815): Create Tablet RPC failed for tablet 6e8c16ee547a46c2ae96a84f02357575: Network error: Client connection negotiation failed: client connection to 127.6.150.1:57815: connect: Connection refused (error 111) I1010 20:58:35.772022 1696 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 6e8c16ee547a46c2ae96a84f02357575 on TS 7160a237fa2e437da8305b75ed2e6ec8 with a delay of 278ms (attempt = 5)... I1010 20:58:35.806377 1686 create-table-itest.cc:83] Waiting for the master to retry creating the tablet 3 times... 2 RPCs seen so far W1010 20:58:35.820402 1695 catalog_manager.cc:2301] TS 75e56306e836413685f502a571284009 (127.6.150.2:38018): Create Tablet RPC failed for tablet 6e8c16ee547a46c2ae96a84f02357575: Network error: Client connection negotiation failed: client connection to 127.6.150.2:38018: connect: Connection refused (error 111) I1010 20:58:35.820417 1695 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 6e8c16ee547a46c2ae96a84f02357575 on TS 75e56306e836413685f502a571284009 with a delay of 279ms (attempt = 5)... I1010 20:58:35.826603 1695 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b1e6a7b25cce46d9a64325794390ab21 on 75e56306e836413685f502a571284009 (127.6.150.2:38018) (Replaced by 6e8c16ee547a46c2ae96a84f02357575 at 2016-10-10 20:58:35 UTC) W1010 20:58:35.826764 1695 catalog_manager.cc:2301] TS 75e56306e836413685f502a571284009 (127.6.150.2:38018): Delete Tablet RPC failed for tablet b1e6a7b25cce46d9a64325794390ab21: Network error: Client connection negotiation failed: client connection to 127.6.150.2:38018: connect: Connection refused (error 111) I1010 20:58:35.826786 1695 catalog_manager.cc:2355] Scheduling retry of b1e6a7b25cce46d9a64325794390ab21 Delete Tablet RPC for TS=75e56306e836413685f502a571284009 with a delay of 263ms (attempt = 5)... I1010 20:58:35.853268 2131 raft_consensus.cc:375] T 6e8c16ee547a46c2ae96a84f02357575 P d09b95f9213649f6ac7f29f92d8e3f11 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:35.853289 2131 raft_consensus.cc:2031] T 6e8c16ee547a46c2ae96a84f02357575 P d09b95f9213649f6ac7f29f92d8e3f11 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:35.853590 2131 raft_consensus.cc:1981] T 6e8c16ee547a46c2ae96a84f02357575 P d09b95f9213649f6ac7f29f92d8e3f11 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.532s I1010 20:58:35.853646 2131 raft_consensus.cc:391] T 6e8c16ee547a46c2ae96a84f02357575 P d09b95f9213649f6ac7f29f92d8e3f11 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e56306e836413685f502a571284009" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 38018 } } peers { permanent_uuid: "7160a237fa2e437da8305b75ed2e6ec8" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 57815 } } peers { permanent_uuid: "d09b95f9213649f6ac7f29f92d8e3f11" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 51431 } } I1010 20:58:35.867913 1696 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b1e6a7b25cce46d9a64325794390ab21 on 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815) (Replaced by 6e8c16ee547a46c2ae96a84f02357575 at 2016-10-10 20:58:35 UTC) W1010 20:58:35.868098 1696 catalog_manager.cc:2301] TS 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815): Delete Tablet RPC failed for tablet b1e6a7b25cce46d9a64325794390ab21: Network error: Client connection negotiation failed: client connection to 127.6.150.1:57815: connect: Connection refused (error 111) I1010 20:58:35.868109 1696 catalog_manager.cc:2355] Scheduling retry of b1e6a7b25cce46d9a64325794390ab21 Delete Tablet RPC for TS=7160a237fa2e437da8305b75ed2e6ec8 with a delay of 268ms (attempt = 5)... I1010 20:58:35.907222 1686 create-table-itest.cc:83] Waiting for the master to retry creating the tablet 3 times... 2 RPCs seen so far I1010 20:58:35.978509 2131 leader_election.cc:223] T 6e8c16ee547a46c2ae96a84f02357575 P d09b95f9213649f6ac7f29f92d8e3f11 [CANDIDATE]: Term 1 election: Requesting vote from peer 75e56306e836413685f502a571284009 I1010 20:58:35.978549 2131 leader_election.cc:223] T 6e8c16ee547a46c2ae96a84f02357575 P d09b95f9213649f6ac7f29f92d8e3f11 [CANDIDATE]: Term 1 election: Requesting vote from peer 7160a237fa2e437da8305b75ed2e6ec8 W1010 20:58:35.979039 1753 leader_election.cc:275] T 6e8c16ee547a46c2ae96a84f02357575 P d09b95f9213649f6ac7f29f92d8e3f11 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 75e56306e836413685f502a571284009: Network error: Client connection negotiation failed: client connection to 127.6.150.2:38018: connect: Connection refused (error 111) W1010 20:58:35.979073 1754 leader_election.cc:275] T 6e8c16ee547a46c2ae96a84f02357575 P d09b95f9213649f6ac7f29f92d8e3f11 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 7160a237fa2e437da8305b75ed2e6ec8: Network error: Client connection negotiation failed: client connection to 127.6.150.1:57815: connect: Connection refused (error 111) I1010 20:58:35.979091 1754 leader_election.cc:248] T 6e8c16ee547a46c2ae96a84f02357575 P d09b95f9213649f6ac7f29f92d8e3f11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:35.979192 2132 raft_consensus.cc:1981] T 6e8c16ee547a46c2ae96a84f02357575 P d09b95f9213649f6ac7f29f92d8e3f11 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.531s I1010 20:58:35.979210 2132 raft_consensus.cc:1823] T 6e8c16ee547a46c2ae96a84f02357575 P d09b95f9213649f6ac7f29f92d8e3f11 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given I1010 20:58:36.008018 1686 create-table-itest.cc:83] Waiting for the master to retry creating the tablet 3 times... 2 RPCs seen so far W1010 20:58:36.050711 1696 catalog_manager.cc:2301] TS 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815): Create Tablet RPC failed for tablet 6e8c16ee547a46c2ae96a84f02357575: Network error: Client connection negotiation failed: client connection to 127.6.150.1:57815: connect: Connection refused (error 111) I1010 20:58:36.050737 1696 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 6e8c16ee547a46c2ae96a84f02357575 on TS 7160a237fa2e437da8305b75ed2e6ec8 with a delay of 436ms (attempt = 6)... I1010 20:58:36.090404 1695 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b1e6a7b25cce46d9a64325794390ab21 on 75e56306e836413685f502a571284009 (127.6.150.2:38018) (Replaced by 6e8c16ee547a46c2ae96a84f02357575 at 2016-10-10 20:58:35 UTC) W1010 20:58:36.090657 1695 catalog_manager.cc:2301] TS 75e56306e836413685f502a571284009 (127.6.150.2:38018): Delete Tablet RPC failed for tablet b1e6a7b25cce46d9a64325794390ab21: Network error: Client connection negotiation failed: client connection to 127.6.150.2:38018: connect: Connection refused (error 111) I1010 20:58:36.090670 1695 catalog_manager.cc:2355] Scheduling retry of b1e6a7b25cce46d9a64325794390ab21 Delete Tablet RPC for TS=75e56306e836413685f502a571284009 with a delay of 545ms (attempt = 6)... W1010 20:58:36.100044 1695 catalog_manager.cc:2301] TS 75e56306e836413685f502a571284009 (127.6.150.2:38018): Create Tablet RPC failed for tablet 6e8c16ee547a46c2ae96a84f02357575: Network error: Client connection negotiation failed: client connection to 127.6.150.2:38018: connect: Connection refused (error 111) I1010 20:58:36.100059 1695 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 6e8c16ee547a46c2ae96a84f02357575 on TS 75e56306e836413685f502a571284009 with a delay of 386ms (attempt = 6)... I1010 20:58:36.108916 1686 create-table-itest.cc:83] Waiting for the master to retry creating the tablet 3 times... 2 RPCs seen so far I1010 20:58:36.136425 1696 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b1e6a7b25cce46d9a64325794390ab21 on 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815) (Replaced by 6e8c16ee547a46c2ae96a84f02357575 at 2016-10-10 20:58:35 UTC) W1010 20:58:36.136577 1696 catalog_manager.cc:2301] TS 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815): Delete Tablet RPC failed for tablet b1e6a7b25cce46d9a64325794390ab21: Network error: Client connection negotiation failed: client connection to 127.6.150.1:57815: connect: Connection refused (error 111) I1010 20:58:36.136596 1696 catalog_manager.cc:2355] Scheduling retry of b1e6a7b25cce46d9a64325794390ab21 Delete Tablet RPC for TS=7160a237fa2e437da8305b75ed2e6ec8 with a delay of 558ms (attempt = 6)... I1010 20:58:36.209733 1686 create-table-itest.cc:83] Waiting for the master to retry creating the tablet 3 times... 2 RPCs seen so far I1010 20:58:36.310614 1686 create-table-itest.cc:83] Waiting for the master to retry creating the tablet 3 times... 2 RPCs seen so far I1010 20:58:36.411401 1686 create-table-itest.cc:83] Waiting for the master to retry creating the tablet 3 times... 2 RPCs seen so far W1010 20:58:36.487020 1695 catalog_manager.cc:2301] TS 75e56306e836413685f502a571284009 (127.6.150.2:38018): Create Tablet RPC failed for tablet 6e8c16ee547a46c2ae96a84f02357575: Network error: Client connection negotiation failed: client connection to 127.6.150.2:38018: connect: Connection refused (error 111) W1010 20:58:36.487040 1695 catalog_manager.cc:2352] Request timed out: CreateTablet RPC for tablet 6e8c16ee547a46c2ae96a84f02357575 on TS 75e56306e836413685f502a571284009 W1010 20:58:36.487926 1696 catalog_manager.cc:2301] TS 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815): Create Tablet RPC failed for tablet 6e8c16ee547a46c2ae96a84f02357575: Network error: Client connection negotiation failed: client connection to 127.6.150.1:57815: connect: Connection refused (error 111) W1010 20:58:36.487943 1696 catalog_manager.cc:2352] Request timed out: CreateTablet RPC for tablet 6e8c16ee547a46c2ae96a84f02357575 on TS 7160a237fa2e437da8305b75ed2e6ec8 W1010 20:58:36.497088 1762 catalog_manager.cc:2975] Tablet 6e8c16ee547a46c2ae96a84f02357575 (table test-table [id=cdc4846c3fe84fb7b1522be60094eca9]) was not created within the allowed timeout. Replacing with a new tablet 16ca418029b24d288343c959bb8f7b50 I1010 20:58:36.497774 1762 catalog_manager.cc:2864] Sending DeleteTablet for 3 replicas of tablet 6e8c16ee547a46c2ae96a84f02357575 I1010 20:58:36.497797 1762 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 6e8c16ee547a46c2ae96a84f02357575 on 75e56306e836413685f502a571284009 (127.6.150.2:38018) (Replaced by 16ca418029b24d288343c959bb8f7b50 at 2016-10-10 20:58:36 UTC) I1010 20:58:36.497822 1762 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 6e8c16ee547a46c2ae96a84f02357575 on 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815) (Replaced by 16ca418029b24d288343c959bb8f7b50 at 2016-10-10 20:58:36 UTC) I1010 20:58:36.497843 1762 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 6e8c16ee547a46c2ae96a84f02357575 on d09b95f9213649f6ac7f29f92d8e3f11 (127.6.150.0:51431) (Replaced by 16ca418029b24d288343c959bb8f7b50 at 2016-10-10 20:58:36 UTC) W1010 20:58:36.497920 1695 catalog_manager.cc:2301] TS 75e56306e836413685f502a571284009 (127.6.150.2:38018): Create Tablet RPC failed for tablet 16ca418029b24d288343c959bb8f7b50: Network error: Client connection negotiation failed: client connection to 127.6.150.2:38018: connect: Connection refused (error 111) I1010 20:58:36.497931 1695 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 16ca418029b24d288343c959bb8f7b50 on TS 75e56306e836413685f502a571284009 with a delay of 19ms (attempt = 1)... W1010 20:58:36.497939 1695 catalog_manager.cc:2301] TS 75e56306e836413685f502a571284009 (127.6.150.2:38018): Delete Tablet RPC failed for tablet 6e8c16ee547a46c2ae96a84f02357575: Network error: Client connection negotiation failed: client connection to 127.6.150.2:38018: connect: Connection refused (error 111) I1010 20:58:36.497943 1695 catalog_manager.cc:2355] Scheduling retry of 6e8c16ee547a46c2ae96a84f02357575 Delete Tablet RPC for TS=75e56306e836413685f502a571284009 with a delay of 42ms (attempt = 1)... I1010 20:58:36.498023 1807 tablet_service.cc:660] Processing DeleteTablet for tablet 6e8c16ee547a46c2ae96a84f02357575 with delete_type TABLET_DATA_DELETED (Replaced by 16ca418029b24d288343c959bb8f7b50 at 2016-10-10 20:58:36 UTC) from {real_user=slave, eff_user=} at 127.0.0.1:43898 I1010 20:58:36.498033 1806 tablet_service.cc:619] Processing CreateTablet for tablet 16ca418029b24d288343c959bb8f7b50 (table=test-table [id=cdc4846c3fe84fb7b1522be60094eca9]), partition=range: [, ) W1010 20:58:36.498039 1696 catalog_manager.cc:2301] TS 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815): Create Tablet RPC failed for tablet 16ca418029b24d288343c959bb8f7b50: Network error: Client connection negotiation failed: client connection to 127.6.150.1:57815: connect: Connection refused (error 111) I1010 20:58:36.498047 1807 tablet_peer.cc:215] Initiating TabletPeer shutdown for tablet: 6e8c16ee547a46c2ae96a84f02357575 I1010 20:58:36.498049 1696 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 16ca418029b24d288343c959bb8f7b50 on TS 7160a237fa2e437da8305b75ed2e6ec8 with a delay of 40ms (attempt = 1)... I1010 20:58:36.498054 1807 maintenance_manager.cc:182] Unregistered op CompactRowSetsOp(6e8c16ee547a46c2ae96a84f02357575) W1010 20:58:36.498055 1696 catalog_manager.cc:2301] TS 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815): Delete Tablet RPC failed for tablet 6e8c16ee547a46c2ae96a84f02357575: Network error: Client connection negotiation failed: client connection to 127.6.150.1:57815: connect: Connection refused (error 111) I1010 20:58:36.498060 1807 maintenance_manager.cc:182] Unregistered op MinorDeltaCompactionOp(6e8c16ee547a46c2ae96a84f02357575) I1010 20:58:36.498060 1696 catalog_manager.cc:2355] Scheduling retry of 6e8c16ee547a46c2ae96a84f02357575 Delete Tablet RPC for TS=7160a237fa2e437da8305b75ed2e6ec8 with a delay of 46ms (attempt = 1)... I1010 20:58:36.498064 1807 maintenance_manager.cc:182] Unregistered op MajorDeltaCompactionOp(6e8c16ee547a46c2ae96a84f02357575) I1010 20:58:36.498073 1807 maintenance_manager.cc:182] Unregistered op FlushMRSOp(6e8c16ee547a46c2ae96a84f02357575) I1010 20:58:36.498077 1807 maintenance_manager.cc:182] Unregistered op FlushDeltaMemStoresOp(6e8c16ee547a46c2ae96a84f02357575) I1010 20:58:36.498080 1807 maintenance_manager.cc:182] Unregistered op LogGCOp(6e8c16ee547a46c2ae96a84f02357575) I1010 20:58:36.498088 1807 raft_consensus.cc:1495] T 6e8c16ee547a46c2ae96a84f02357575 P d09b95f9213649f6ac7f29f92d8e3f11 [term 1 FOLLOWER]: Raft consensus shutting down. I1010 20:58:36.498095 1807 raft_consensus.cc:1511] T 6e8c16ee547a46c2ae96a84f02357575 P d09b95f9213649f6ac7f29f92d8e3f11 [term 1 FOLLOWER]: Raft consensus is shut down! I1010 20:58:36.498446 1807 ts_tablet_manager.cc:916] T 6e8c16ee547a46c2ae96a84f02357575 P d09b95f9213649f6ac7f29f92d8e3f11: Deleting tablet data with delete state TABLET_DATA_DELETED I1010 20:58:36.498625 1806 ts_tablet_manager.cc:766] Registered tablet 16ca418029b24d288343c959bb8f7b50 I1010 20:58:36.498888 2138 ts_tablet_manager.cc:632] T 16ca418029b24d288343c959bb8f7b50 P d09b95f9213649f6ac7f29f92d8e3f11: Bootstrapping tablet I1010 20:58:36.499022 2138 tablet_bootstrap.cc:380] T 16ca418029b24d288343c959bb8f7b50 P d09b95f9213649f6ac7f29f92d8e3f11: Bootstrap starting. I1010 20:58:36.499147 2138 tablet_bootstrap.cc:542] T 16ca418029b24d288343c959bb8f7b50 P d09b95f9213649f6ac7f29f92d8e3f11: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:36.499243 2138 tablet_bootstrap.cc:482] T 16ca418029b24d288343c959bb8f7b50 P d09b95f9213649f6ac7f29f92d8e3f11: No blocks or log segments found. Creating new log. I1010 20:58:36.499755 2138 tablet_bootstrap.cc:380] T 16ca418029b24d288343c959bb8f7b50 P d09b95f9213649f6ac7f29f92d8e3f11: No bootstrap required, opened a new log I1010 20:58:36.499830 2138 ts_tablet_manager.cc:637] T 16ca418029b24d288343c959bb8f7b50 P d09b95f9213649f6ac7f29f92d8e3f11: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:36.500202 2138 raft_consensus.cc:275] T 16ca418029b24d288343c959bb8f7b50 P d09b95f9213649f6ac7f29f92d8e3f11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7160a237fa2e437da8305b75ed2e6ec8" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 57815 } } peers { permanent_uuid: "75e56306e836413685f502a571284009" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 38018 } } peers { permanent_uuid: "d09b95f9213649f6ac7f29f92d8e3f11" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 51431 } } I1010 20:58:36.500221 2138 raft_consensus.cc:306] T 16ca418029b24d288343c959bb8f7b50 P d09b95f9213649f6ac7f29f92d8e3f11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:36.500228 2138 raft_consensus.cc:502] T 16ca418029b24d288343c959bb8f7b50 P d09b95f9213649f6ac7f29f92d8e3f11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09b95f9213649f6ac7f29f92d8e3f11, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:36.500236 2138 consensus_queue.cc:168] T 16ca418029b24d288343c959bb8f7b50 P d09b95f9213649f6ac7f29f92d8e3f11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:36.500449 2138 ts_tablet_manager.cc:664] T 16ca418029b24d288343c959bb8f7b50 P d09b95f9213649f6ac7f29f92d8e3f11: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:36.512137 1686 create-table-itest.cc:83] Waiting for the master to retry creating the tablet 3 times... 3 RPCs seen so far I1010 20:58:36.512473 1686 external_mini_cluster.cc:586] Running /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-1 --fs_data_dirs=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-1 --rpc_bind_addresses=127.6.150.1:57815 --local_ip_for_outbound_sockets=127.6.150.1 --webserver_port=37972 --webserver_interface=127.6.150.1 --tserver_master_addrs=127.0.0.1:46501 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-1 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-1/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 --unlock_experimental_flags --unlock_unsafe_flags W1010 20:58:36.517396 1695 catalog_manager.cc:2301] TS 75e56306e836413685f502a571284009 (127.6.150.2:38018): Create Tablet RPC failed for tablet 16ca418029b24d288343c959bb8f7b50: Network error: Client connection negotiation failed: client connection to 127.6.150.2:38018: connect: Connection refused (error 111) I1010 20:58:36.517416 1695 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 16ca418029b24d288343c959bb8f7b50 on TS 75e56306e836413685f502a571284009 with a delay of 75ms (attempt = 2)... WARNING: Logging before InitGoogleLogging() is written to STDERR W1010 20:58:36.517699 2143 flags.cc:294] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1010 20:58:36.517771 2143 flags.cc:294] Enabled unsafe flag: --never_fsync=true W1010 20:58:36.518242 2143 flags.cc:294] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.150.1 I1010 20:58:36.518343 2143 mem_tracker.cc:140] MemTracker: hard memory limit is 11.777661 GB I1010 20:58:36.518373 2143 mem_tracker.cc:142] MemTracker: soft memory limit is 7.066597 GB I1010 20:58:36.519654 2143 tablet_server_main.cc:54] Initializing tablet server... I1010 20:58:36.519804 2143 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 969115 I1010 20:58:36.521178 2143 fs_manager.cc:243] Opened local filesystem: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-1 uuid: "7160a237fa2e437da8305b75ed2e6ec8" format_stamp: "Formatted at 2016-10-10 20:58:34 on dist-test-slave-dist-test-slave-ymsa" I1010 20:58:36.523795 2143 tablet_server_main.cc:57] Starting tablet server... I1010 20:58:36.531750 2143 rpc_server.cc:164] RPC server started. Bound to: 127.6.150.1:57815 I1010 20:58:36.531805 2143 webserver.cc:125] Starting webserver on 127.6.150.1:37972 I1010 20:58:36.531821 2143 webserver.cc:136] Document root disabled I1010 20:58:36.531949 2143 webserver.cc:220] Webserver started. Bound to: http://127.6.150.1:37972/ I1010 20:58:36.532073 2143 server_base.cc:239] Dumped server information to /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-1/info.pb I1010 20:58:36.532318 2143 tablet_server_main.cc:60] Tablet server successfully started. I1010 20:58:36.532629 1807 ts_tablet_manager.cc:926] T 6e8c16ee547a46c2ae96a84f02357575 P d09b95f9213649f6ac7f29f92d8e3f11: Tablet deleted. Last logged OpId: 0.0 I1010 20:58:36.533069 1697 catalog_manager.cc:2567] TS d09b95f9213649f6ac7f29f92d8e3f11 (127.6.150.0:51431): tablet 6e8c16ee547a46c2ae96a84f02357575 (table test-table [id=cdc4846c3fe84fb7b1522be60094eca9]) successfully deleted I1010 20:58:36.533089 1686 external_mini_cluster.cc:622] Started /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver as pid 2143 I1010 20:58:36.533192 1686 external_mini_cluster.cc:586] Running /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-2 --fs_data_dirs=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-2 --rpc_bind_addresses=127.6.150.2:38018 --local_ip_for_outbound_sockets=127.6.150.2 --webserver_port=36092 --webserver_interface=127.6.150.2 --tserver_master_addrs=127.0.0.1:46501 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-2 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-2/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 --unlock_experimental_flags --unlock_unsafe_flags I1010 20:58:36.533344 2257 heartbeater.cc:292] Connected to a master server at 127.0.0.1:46501 I1010 20:58:36.533362 2257 heartbeater.cc:359] Registering TS with master... I1010 20:58:36.533377 2257 heartbeater.cc:373] Master 127.0.0.1:46501 requested a full tablet report, sending... I1010 20:58:36.533501 1712 ts_manager.cc:82] Re-registered known tserver with Master: 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815) WARNING: Logging before InitGoogleLogging() is written to STDERR W1010 20:58:36.537796 2260 flags.cc:294] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1010 20:58:36.537863 2260 flags.cc:294] Enabled unsafe flag: --never_fsync=true W1010 20:58:36.538328 2260 flags.cc:294] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.150.2 I1010 20:58:36.538429 2260 mem_tracker.cc:140] MemTracker: hard memory limit is 11.777661 GB I1010 20:58:36.538452 2260 mem_tracker.cc:142] MemTracker: soft memory limit is 7.066597 GB I1010 20:58:36.539175 2194 tablet_service.cc:619] Processing CreateTablet for tablet 16ca418029b24d288343c959bb8f7b50 (table=test-table [id=cdc4846c3fe84fb7b1522be60094eca9]), partition=range: [, ) I1010 20:58:36.539696 2260 tablet_server_main.cc:54] Initializing tablet server... I1010 20:58:36.539840 2260 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 969115 I1010 20:58:36.539933 2194 ts_tablet_manager.cc:766] Registered tablet 16ca418029b24d288343c959bb8f7b50 I1010 20:58:36.540751 1695 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 6e8c16ee547a46c2ae96a84f02357575 on 75e56306e836413685f502a571284009 (127.6.150.2:38018) (Replaced by 16ca418029b24d288343c959bb8f7b50 at 2016-10-10 20:58:36 UTC) W1010 20:58:36.540899 1695 catalog_manager.cc:2301] TS 75e56306e836413685f502a571284009 (127.6.150.2:38018): Delete Tablet RPC failed for tablet 6e8c16ee547a46c2ae96a84f02357575: Network error: Client connection negotiation failed: client connection to 127.6.150.2:38018: connect: Connection refused (error 111) I1010 20:58:36.540913 1695 catalog_manager.cc:2355] Scheduling retry of 6e8c16ee547a46c2ae96a84f02357575 Delete Tablet RPC for TS=75e56306e836413685f502a571284009 with a delay of 67ms (attempt = 2)... I1010 20:58:36.541013 2263 ts_tablet_manager.cc:632] T 16ca418029b24d288343c959bb8f7b50 P 7160a237fa2e437da8305b75ed2e6ec8: Bootstrapping tablet I1010 20:58:36.541187 2263 tablet_bootstrap.cc:380] T 16ca418029b24d288343c959bb8f7b50 P 7160a237fa2e437da8305b75ed2e6ec8: Bootstrap starting. I1010 20:58:36.541219 2260 fs_manager.cc:243] Opened local filesystem: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-2 uuid: "75e56306e836413685f502a571284009" format_stamp: "Formatted at 2016-10-10 20:58:34 on dist-test-slave-dist-test-slave-ymsa" I1010 20:58:36.541352 2263 tablet_bootstrap.cc:542] T 16ca418029b24d288343c959bb8f7b50 P 7160a237fa2e437da8305b75ed2e6ec8: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:36.541494 2263 tablet_bootstrap.cc:482] T 16ca418029b24d288343c959bb8f7b50 P 7160a237fa2e437da8305b75ed2e6ec8: No blocks or log segments found. Creating new log. I1010 20:58:36.541631 2263 log.cc:347] Log is configured to *not* fsync() on all Append() calls I1010 20:58:36.543006 2263 tablet_bootstrap.cc:380] T 16ca418029b24d288343c959bb8f7b50 P 7160a237fa2e437da8305b75ed2e6ec8: No bootstrap required, opened a new log I1010 20:58:36.543032 2263 ts_tablet_manager.cc:637] T 16ca418029b24d288343c959bb8f7b50 P 7160a237fa2e437da8305b75ed2e6ec8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s I1010 20:58:36.543416 2263 raft_consensus.cc:275] T 16ca418029b24d288343c959bb8f7b50 P 7160a237fa2e437da8305b75ed2e6ec8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7160a237fa2e437da8305b75ed2e6ec8" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 57815 } } peers { permanent_uuid: "75e56306e836413685f502a571284009" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 38018 } } peers { permanent_uuid: "d09b95f9213649f6ac7f29f92d8e3f11" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 51431 } } I1010 20:58:36.543443 2263 raft_consensus.cc:306] T 16ca418029b24d288343c959bb8f7b50 P 7160a237fa2e437da8305b75ed2e6ec8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:36.543457 2263 raft_consensus.cc:502] T 16ca418029b24d288343c959bb8f7b50 P 7160a237fa2e437da8305b75ed2e6ec8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7160a237fa2e437da8305b75ed2e6ec8, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:36.543473 2263 consensus_queue.cc:168] T 16ca418029b24d288343c959bb8f7b50 P 7160a237fa2e437da8305b75ed2e6ec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:36.543604 2257 heartbeater.cc:365] Master 127.0.0.1:46501 was elected leader, sending a full tablet report... I1010 20:58:36.543805 2263 ts_tablet_manager.cc:664] T 16ca418029b24d288343c959bb8f7b50 P 7160a237fa2e437da8305b75ed2e6ec8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:36.544085 2260 tablet_server_main.cc:57] Starting tablet server... I1010 20:58:36.544260 1696 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 6e8c16ee547a46c2ae96a84f02357575 on 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815) (Replaced by 16ca418029b24d288343c959bb8f7b50 at 2016-10-10 20:58:36 UTC) I1010 20:58:36.544348 2194 tablet_service.cc:660] Processing DeleteTablet for tablet 6e8c16ee547a46c2ae96a84f02357575 with delete_type TABLET_DATA_DELETED (Replaced by 16ca418029b24d288343c959bb8f7b50 at 2016-10-10 20:58:36 UTC) from {real_user=slave, eff_user=} at 127.0.0.1:40792 W1010 20:58:36.544425 1696 catalog_manager.cc:2545] TS 7160a237fa2e437da8305b75ed2e6ec8 (127.6.150.1:57815): delete failed for tablet 6e8c16ee547a46c2ae96a84f02357575 because the tablet was not found. No further retry: Not found: Tablet not found: 6e8c16ee547a46c2ae96a84f02357575 I1010 20:58:36.553596 2260 rpc_server.cc:164] RPC server started. Bound to: 127.6.150.2:38018 I1010 20:58:36.553637 2260 webserver.cc:125] Starting webserver on 127.6.150.2:36092 I1010 20:58:36.553649 2260 webserver.cc:136] Document root disabled I1010 20:58:36.553741 2260 webserver.cc:220] Webserver started. Bound to: http://127.6.150.2:36092/ I1010 20:58:36.553848 2260 server_base.cc:239] Dumped server information to /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation.1476133114077700-1686/minicluster-data/ts-2/info.pb I1010 20:58:36.553907 1686 external_mini_cluster.cc:622] Started /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver as pid 2260 I1010 20:58:36.553964 1686 create-table-itest.cc:112] Waiting for only one tablet to be left on TS 0. Currently have: I1010 20:58:36.554126 2260 tablet_server_main.cc:60] Tablet server successfully started. I1010 20:58:36.554847 2379 heartbeater.cc:292] Connected to a master server at 127.0.0.1:46501 I1010 20:58:36.554862 2379 heartbeater.cc:359] Registering TS with master... I1010 20:58:36.554872 2379 heartbeater.cc:373] Master 127.0.0.1:46501 requested a full tablet report, sending... I1010 20:58:36.554998 1712 ts_manager.cc:82] Re-registered known tserver with Master: 75e56306e836413685f502a571284009 (127.6.150.2:38018) I1010 20:58:36.594208 2316 tablet_service.cc:619] Processing CreateTablet for tablet 16ca418029b24d288343c959bb8f7b50 (table=test-table [id=cdc4846c3fe84fb7b1522be60094eca9]), partition=range: [, ) I1010 20:58:36.594961 2316 ts_tablet_manager.cc:766] Registered tablet 16ca418029b24d288343c959bb8f7b50 I1010 20:58:36.595111 2382 ts_tablet_manager.cc:632] T 16ca418029b24d288343c959bb8f7b50 P 75e56306e836413685f502a571284009: Bootstrapping tablet I1010 20:58:36.595263 2382 tablet_bootstrap.cc:380] T 16ca418029b24d288343c959bb8f7b50 P 75e56306e836413685f502a571284009: Bootstrap starting. I1010 20:58:36.595399 2382 tablet_bootstrap.cc:542] T 16ca418029b24d288343c959bb8f7b50 P 75e56306e836413685f502a571284009: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:36.595506 2382 tablet_bootstrap.cc:482] T 16ca418029b24d288343c959bb8f7b50 P 75e56306e836413685f502a571284009: No blocks or log segments found. Creating new log. I1010 20:58:36.595631 2382 log.cc:347] Log is configured to *not* fsync() on all Append() calls I1010 20:58:36.596035 2382 tablet_bootstrap.cc:380] T 16ca418029b24d288343c959bb8f7b50 P 75e56306e836413685f502a571284009: No bootstrap required, opened a new log I1010 20:58:36.596055 2382 ts_tablet_manager.cc:637] T 16ca418029b24d288343c959bb8f7b50 P 75e56306e836413685f502a571284009: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:36.596366 2382 raft_consensus.cc:275] T 16ca418029b24d288343c959bb8f7b50 P 75e56306e836413685f502a571284009 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7160a237fa2e437da8305b75ed2e6ec8" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 57815 } } peers { permanent_uuid: "75e56306e836413685f502a571284009" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 38018 } } peers { permanent_uuid: "d09b95f9213649f6ac7f29f92d8e3f11" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 51431 } } I1010 20:58:36.596387 2382 raft_consensus.cc:306] T 16ca418029b24d288343c959bb8f7b50 P 75e56306e836413685f502a571284009 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:36.596400 2382 raft_consensus.cc:502] T 16ca418029b24d288343c959bb8f7b50 P 75e56306e836413685f502a571284009 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75e56306e836413685f502a571284009, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:36.596411 2382 consensus_queue.cc:168] T 16ca418029b24d288343c959bb8f7b50 P 75e56306e836413685f502a571284009 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:36.596586 2379 heartbeater.cc:365] Master 127.0.0.1:46501 was elected leader, sending a full tablet report... I1010 20:58:36.596587 2382 ts_tablet_manager.cc:664] T 16ca418029b24d288343c959bb8f7b50 P 75e56306e836413685f502a571284009: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:36.608551 1695 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 6e8c16ee547a46c2ae96a84f02357575 on 75e56306e836413685f502a571284009 (127.6.150.2:38018) (Replaced by 16ca418029b24d288343c959bb8f7b50 at 2016-10-10 20:58:36 UTC) I1010 20:58:36.608660 2316 tablet_service.cc:660] Processing DeleteTablet for tablet 6e8c16ee547a46c2ae96a84f02357575 with delete_type TABLET_DATA_DELETED (Replaced by 16ca418029b24d288343c959bb8f7b50 at 2016-10-10 20:58:36 UTC) from {real_user=slave, eff_user=} at 127.0.0.1:34203 W1010 20:58:36.608752 1695 catalog_manager.cc:2545] TS 75e56306e836413685f502a571284009 (127.6.150.2:38018): delete failed for tablet 6e8c16ee547a46c2ae96a84f02357575 because the tablet was not found. No further retry: Not found: Tablet not found: 6e8c16ee547a46c2ae96a84f02357575 I1010 20:58:36.635920 1695 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b1e6a7b25cce46d9a64325794390ab21 on 75e56306e836413685f502a571284009 (127.6.150.2:38018) (Replaced by 6e8c16ee547a46c2ae96a84f02357575 at 2016-10-10 20:58:35 UTC) I1010 20:58:36.636016 2316 tablet_service.cc:660] Processing DeleteTablet for tablet b1e6a7b25cce46d9a64325794390ab21 with delete_type TABLET_DATA_DELETED (Replaced by 6e8c16ee547a46c2ae96a84f02357575 at 2016-10-10 20:58:35 UTC) from {real_user=slave, eff_user=} at 127.0.0.1:34203 W1010 20:58:36.636137 1695 catalog_manager.cc:2545] TS 75e56306e836413685f502a571284009 (127.6.150.2:38018): delete failed for tablet b1e6a7b25cce46d9a64325794390ab21 because the tablet was not found. No further retry: Not found: Tablet not found: b1e6a7b25cce46d9a64325794390ab21 I1010 20:58:36.654145 1686 external_mini_cluster.cc:687] Killing /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver with pid 1749 I1010 20:58:36.656051 1686 external_mini_cluster.cc:687] Killing /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver with pid 2143 I1010 20:58:36.657376 1686 external_mini_cluster.cc:687] Killing /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver with pid 2260 I1010 20:58:36.658841 1686 external_mini_cluster.cc:687] Killing /tmp/run_tha_testXiwMYW/build/release/bin/kudu-master with pid 1691 [ OK ] CreateTableITest.TestCreateWhenMajorityOfReplicasFailCreation (2582 ms) [ RUN ] CreateTableITest.TestSpreadReplicasEvenly I1010 20:58:36.663610 1686 external_mini_cluster.cc:586] Running /tmp/run_tha_testXiwMYW/build/release/bin/kudu-master kudu-master --fs_wal_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/master-0 --fs_data_dirs=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/master-0 --rpc_bind_addresses=127.0.0.1:0 --webserver_interface=localhost --webserver_port=0 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/master-0 --server_dump_info_path=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/master-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 --unlock_experimental_flags --unlock_unsafe_flags WARNING: Logging before InitGoogleLogging() is written to STDERR W1010 20:58:36.668830 2388 flags.cc:294] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true I1010 20:58:36.669510 2388 mem_tracker.cc:140] MemTracker: hard memory limit is 11.777661 GB I1010 20:58:36.669525 2388 mem_tracker.cc:142] MemTracker: soft memory limit is 7.066597 GB I1010 20:58:36.670663 2388 master_main.cc:58] Initializing master server... I1010 20:58:36.670881 2388 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 969115 I1010 20:58:36.671703 2388 server_base.cc:167] Could not load existing FS layout: Not found: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/master-0/instance: No such file or directory (error 2) I1010 20:58:36.671715 2388 server_base.cc:168] Creating new FS layout I1010 20:58:37.008450 2388 fs_manager.cc:354] Generated new instance metadata in path /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/master-0/instance: uuid: "89d5c63c41ed40f5b968d37d123c09c4" format_stamp: "Formatted at 2016-10-10 20:58:36 on dist-test-slave-dist-test-slave-ymsa" I1010 20:58:38.647297 2388 fs_manager.cc:243] Opened local filesystem: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/master-0 uuid: "89d5c63c41ed40f5b968d37d123c09c4" format_stamp: "Formatted at 2016-10-10 20:58:36 on dist-test-slave-dist-test-slave-ymsa" I1010 20:58:38.649662 2388 master_main.cc:61] Starting Master server... I1010 20:58:38.653467 2388 rpc_server.cc:164] RPC server started. Bound to: 127.0.0.1:51244 I1010 20:58:38.653508 2388 webserver.cc:125] Starting webserver on localhost:0 I1010 20:58:38.653514 2388 webserver.cc:136] Document root disabled I1010 20:58:38.653694 2388 net_util.cc:195] Address 127.0.0.1:0 for localhost:0 duplicates an earlier resolved entry. I1010 20:58:38.653791 2388 webserver.cc:220] Webserver started. Bound to: http://127.0.0.1:43824/ I1010 20:58:38.653898 2388 server_base.cc:239] Dumped server information to /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/master-0/info.pb I1010 20:58:38.657033 1686 external_mini_cluster.cc:622] Started /tmp/run_tha_testXiwMYW/build/release/bin/kudu-master as pid 2388 I1010 20:58:38.657097 1686 external_mini_cluster.cc:586] Running /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-0 --fs_data_dirs=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-0 --rpc_bind_addresses=127.6.150.0:0 --local_ip_for_outbound_sockets=127.6.150.0 --webserver_interface=127.6.150.0 --webserver_port=0 --tserver_master_addrs=127.0.0.1:51244 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-0 --never_fsync --never_fsync --server_dump_info_path=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 --unlock_experimental_flags --unlock_unsafe_flags WARNING: Logging before InitGoogleLogging() is written to STDERR W1010 20:58:38.661711 2443 flags.cc:294] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1010 20:58:38.661782 2443 flags.cc:294] Enabled unsafe flag: --never_fsync=true W1010 20:58:38.662166 2443 flags.cc:294] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.150.0 I1010 20:58:38.662252 2443 mem_tracker.cc:140] MemTracker: hard memory limit is 11.777661 GB I1010 20:58:38.662263 2443 mem_tracker.cc:142] MemTracker: soft memory limit is 7.066597 GB I1010 20:58:38.663264 2443 tablet_server_main.cc:54] Initializing tablet server... I1010 20:58:38.663401 2443 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 970115 I1010 20:58:38.663981 2443 server_base.cc:167] Could not load existing FS layout: Not found: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-0/instance: No such file or directory (error 2) I1010 20:58:38.663992 2443 server_base.cc:168] Creating new FS layout I1010 20:58:38.664947 2443 fs_manager.cc:354] Generated new instance metadata in path /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-0/instance: uuid: "84bde34331b94c4ea003e48f9317f792" format_stamp: "Formatted at 2016-10-10 20:58:38 on dist-test-slave-dist-test-slave-ymsa" I1010 20:58:38.665524 2443 fs_manager.cc:243] Opened local filesystem: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-0 uuid: "84bde34331b94c4ea003e48f9317f792" format_stamp: "Formatted at 2016-10-10 20:58:38 on dist-test-slave-dist-test-slave-ymsa" I1010 20:58:38.882259 2443 tablet_server_main.cc:57] Starting tablet server... I1010 20:58:38.890432 2443 rpc_server.cc:164] RPC server started. Bound to: 127.6.150.0:40113 I1010 20:58:38.890502 2443 webserver.cc:125] Starting webserver on 127.6.150.0:0 I1010 20:58:38.890508 2443 webserver.cc:136] Document root disabled I1010 20:58:38.890635 2443 webserver.cc:220] Webserver started. Bound to: http://127.6.150.0:49931/ I1010 20:58:38.890777 2443 server_base.cc:239] Dumped server information to /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-0/info.pb I1010 20:58:38.891008 2443 tablet_server_main.cc:60] Tablet server successfully started. I1010 20:58:38.891810 2557 heartbeater.cc:292] Connected to a master server at 127.0.0.1:51244 I1010 20:58:38.891827 2557 heartbeater.cc:359] Registering TS with master... I1010 20:58:38.891836 2557 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:38.892045 2557 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:58:38.892073 2557 heartbeater.cc:359] Registering TS with master... I1010 20:58:38.892079 2557 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... I1010 20:58:38.892199 1686 external_mini_cluster.cc:622] Started /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver as pid 2443 W1010 20:58:38.892238 2557 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:58:38.892253 2557 heartbeater.cc:359] Registering TS with master... I1010 20:58:38.892259 1686 external_mini_cluster.cc:586] Running /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-1 --fs_data_dirs=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-1 --rpc_bind_addresses=127.6.150.1:0 --local_ip_for_outbound_sockets=127.6.150.1 --webserver_interface=127.6.150.1 --webserver_port=0 --tserver_master_addrs=127.0.0.1:51244 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-1 --never_fsync --never_fsync --server_dump_info_path=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-1/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 --unlock_experimental_flags --unlock_unsafe_flags I1010 20:58:38.892266 2557 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:38.892431 2557 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 W1010 20:58:38.892444 2557 heartbeater.cc:323] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. WARNING: Logging before InitGoogleLogging() is written to STDERR W1010 20:58:38.896471 2561 flags.cc:294] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1010 20:58:38.896541 2561 flags.cc:294] Enabled unsafe flag: --never_fsync=true W1010 20:58:38.897116 2561 flags.cc:294] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.150.1 I1010 20:58:38.897223 2561 mem_tracker.cc:140] MemTracker: hard memory limit is 11.777661 GB I1010 20:58:38.897233 2561 mem_tracker.cc:142] MemTracker: soft memory limit is 7.066597 GB I1010 20:58:38.898082 2561 tablet_server_main.cc:54] Initializing tablet server... I1010 20:58:38.898192 2561 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 970115 I1010 20:58:38.898813 2561 server_base.cc:167] Could not load existing FS layout: Not found: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-1/instance: No such file or directory (error 2) I1010 20:58:38.898824 2561 server_base.cc:168] Creating new FS layout I1010 20:58:38.899734 2561 fs_manager.cc:354] Generated new instance metadata in path /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-1/instance: uuid: "1031512f5120422e9e22a0afabd78f61" format_stamp: "Formatted at 2016-10-10 20:58:38 on dist-test-slave-dist-test-slave-ymsa" I1010 20:58:38.900254 2561 fs_manager.cc:243] Opened local filesystem: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-1 uuid: "1031512f5120422e9e22a0afabd78f61" format_stamp: "Formatted at 2016-10-10 20:58:38 on dist-test-slave-dist-test-slave-ymsa" I1010 20:58:38.902379 2561 tablet_server_main.cc:57] Starting tablet server... I1010 20:58:38.910958 2561 rpc_server.cc:164] RPC server started. Bound to: 127.6.150.1:46963 I1010 20:58:38.910991 2561 webserver.cc:125] Starting webserver on 127.6.150.1:0 I1010 20:58:38.910996 2561 webserver.cc:136] Document root disabled I1010 20:58:38.911088 2561 webserver.cc:220] Webserver started. Bound to: http://127.6.150.1:43398/ I1010 20:58:39.259747 2561 server_base.cc:239] Dumped server information to /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-1/info.pb I1010 20:58:39.260045 2561 tablet_server_main.cc:60] Tablet server successfully started. I1010 20:58:39.260875 2675 heartbeater.cc:292] Connected to a master server at 127.0.0.1:51244 I1010 20:58:39.260903 2675 heartbeater.cc:359] Registering TS with master... I1010 20:58:39.260915 2675 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:39.261070 2675 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:58:39.261085 2675 heartbeater.cc:359] Registering TS with master... I1010 20:58:39.261095 2675 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:39.261253 2675 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:58:39.261265 2675 heartbeater.cc:359] Registering TS with master... I1010 20:58:39.261274 2675 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:39.261414 2675 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 W1010 20:58:39.261425 2675 heartbeater.cc:323] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. I1010 20:58:39.269842 1686 external_mini_cluster.cc:622] Started /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver as pid 2561 I1010 20:58:39.269937 1686 external_mini_cluster.cc:586] Running /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-2 --fs_data_dirs=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-2 --rpc_bind_addresses=127.6.150.2:0 --local_ip_for_outbound_sockets=127.6.150.2 --webserver_interface=127.6.150.2 --webserver_port=0 --tserver_master_addrs=127.0.0.1:51244 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-2 --never_fsync --never_fsync --server_dump_info_path=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-2/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 --unlock_experimental_flags --unlock_unsafe_flags WARNING: Logging before InitGoogleLogging() is written to STDERR W1010 20:58:39.275108 2678 flags.cc:294] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1010 20:58:39.275195 2678 flags.cc:294] Enabled unsafe flag: --never_fsync=true W1010 20:58:39.275630 2678 flags.cc:294] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.150.2 I1010 20:58:39.275759 2678 mem_tracker.cc:140] MemTracker: hard memory limit is 11.777661 GB I1010 20:58:39.275770 2678 mem_tracker.cc:142] MemTracker: soft memory limit is 7.066597 GB I1010 20:58:39.277014 2678 tablet_server_main.cc:54] Initializing tablet server... I1010 20:58:39.277189 2678 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 970615 I1010 20:58:39.277946 2678 server_base.cc:167] Could not load existing FS layout: Not found: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-2/instance: No such file or directory (error 2) I1010 20:58:39.277957 2678 server_base.cc:168] Creating new FS layout I1010 20:58:39.278925 2678 fs_manager.cc:354] Generated new instance metadata in path /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-2/instance: uuid: "f6d15ccd64ec49be8396ed78e2a88377" format_stamp: "Formatted at 2016-10-10 20:58:39 on dist-test-slave-dist-test-slave-ymsa" I1010 20:58:39.279500 2678 fs_manager.cc:243] Opened local filesystem: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-2 uuid: "f6d15ccd64ec49be8396ed78e2a88377" format_stamp: "Formatted at 2016-10-10 20:58:39 on dist-test-slave-dist-test-slave-ymsa" I1010 20:58:39.281746 2678 tablet_server_main.cc:57] Starting tablet server... I1010 20:58:39.290776 2678 rpc_server.cc:164] RPC server started. Bound to: 127.6.150.2:33554 I1010 20:58:39.290813 2678 webserver.cc:125] Starting webserver on 127.6.150.2:0 I1010 20:58:39.290820 2678 webserver.cc:136] Document root disabled I1010 20:58:39.290906 2678 webserver.cc:220] Webserver started. Bound to: http://127.6.150.2:37434/ I1010 20:58:39.291002 2678 server_base.cc:239] Dumped server information to /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-2/info.pb I1010 20:58:39.291309 2678 tablet_server_main.cc:60] Tablet server successfully started. I1010 20:58:39.291998 2792 heartbeater.cc:292] Connected to a master server at 127.0.0.1:51244 I1010 20:58:39.292016 2792 heartbeater.cc:359] Registering TS with master... I1010 20:58:39.292024 2792 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:39.292173 2792 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:58:39.292188 2792 heartbeater.cc:359] Registering TS with master... I1010 20:58:39.292197 2792 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:39.292343 2792 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:58:39.292357 2792 heartbeater.cc:359] Registering TS with master... I1010 20:58:39.292366 2792 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:39.292501 2792 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 W1010 20:58:39.292513 2792 heartbeater.cc:323] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. I1010 20:58:39.300621 1686 external_mini_cluster.cc:622] Started /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver as pid 2678 I1010 20:58:39.300669 1686 external_mini_cluster.cc:586] Running /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-3 --fs_data_dirs=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-3 --rpc_bind_addresses=127.6.150.3:0 --local_ip_for_outbound_sockets=127.6.150.3 --webserver_interface=127.6.150.3 --webserver_port=0 --tserver_master_addrs=127.0.0.1:51244 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-3 --never_fsync --never_fsync --server_dump_info_path=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-3/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 --unlock_experimental_flags --unlock_unsafe_flags WARNING: Logging before InitGoogleLogging() is written to STDERR W1010 20:58:39.304585 2795 flags.cc:294] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1010 20:58:39.304654 2795 flags.cc:294] Enabled unsafe flag: --never_fsync=true W1010 20:58:39.305007 2795 flags.cc:294] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.150.3 I1010 20:58:39.305083 2795 mem_tracker.cc:140] MemTracker: hard memory limit is 11.777661 GB I1010 20:58:39.305091 2795 mem_tracker.cc:142] MemTracker: soft memory limit is 7.066597 GB I1010 20:58:39.306186 2795 tablet_server_main.cc:54] Initializing tablet server... I1010 20:58:39.306350 2795 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 970615 I1010 20:58:39.306974 2795 server_base.cc:167] Could not load existing FS layout: Not found: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-3/instance: No such file or directory (error 2) I1010 20:58:39.306985 2795 server_base.cc:168] Creating new FS layout I1010 20:58:39.307979 2795 fs_manager.cc:354] Generated new instance metadata in path /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-3/instance: uuid: "8de6cfa864d64c58bf4eafb689517c93" format_stamp: "Formatted at 2016-10-10 20:58:39 on dist-test-slave-dist-test-slave-ymsa" I1010 20:58:39.308569 2795 fs_manager.cc:243] Opened local filesystem: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-3 uuid: "8de6cfa864d64c58bf4eafb689517c93" format_stamp: "Formatted at 2016-10-10 20:58:39 on dist-test-slave-dist-test-slave-ymsa" I1010 20:58:39.310839 2795 tablet_server_main.cc:57] Starting tablet server... I1010 20:58:39.319290 2795 rpc_server.cc:164] RPC server started. Bound to: 127.6.150.3:40117 I1010 20:58:39.319329 2795 webserver.cc:125] Starting webserver on 127.6.150.3:0 I1010 20:58:39.319350 2795 webserver.cc:136] Document root disabled I1010 20:58:39.319438 2795 webserver.cc:220] Webserver started. Bound to: http://127.6.150.3:37160/ I1010 20:58:39.319527 2795 server_base.cc:239] Dumped server information to /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-3/info.pb I1010 20:58:39.319741 2795 tablet_server_main.cc:60] Tablet server successfully started. I1010 20:58:39.320363 2909 heartbeater.cc:292] Connected to a master server at 127.0.0.1:51244 I1010 20:58:39.320376 2909 heartbeater.cc:359] Registering TS with master... I1010 20:58:39.320386 2909 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:39.320569 2909 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:58:39.320580 2909 heartbeater.cc:359] Registering TS with master... I1010 20:58:39.320587 2909 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:39.320735 2909 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:58:39.320746 2909 heartbeater.cc:359] Registering TS with master... I1010 20:58:39.320754 2909 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:39.320888 2909 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 W1010 20:58:39.320899 2909 heartbeater.cc:323] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. I1010 20:58:39.321209 1686 external_mini_cluster.cc:622] Started /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver as pid 2795 I1010 20:58:39.321257 1686 external_mini_cluster.cc:586] Running /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-4 --fs_data_dirs=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-4 --rpc_bind_addresses=127.6.150.4:0 --local_ip_for_outbound_sockets=127.6.150.4 --webserver_interface=127.6.150.4 --webserver_port=0 --tserver_master_addrs=127.0.0.1:51244 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-4 --never_fsync --never_fsync --server_dump_info_path=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-4/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 --unlock_experimental_flags --unlock_unsafe_flags WARNING: Logging before InitGoogleLogging() is written to STDERR W1010 20:58:39.325240 2912 flags.cc:294] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1010 20:58:39.325299 2912 flags.cc:294] Enabled unsafe flag: --never_fsync=true W1010 20:58:39.325677 2912 flags.cc:294] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.150.4 I1010 20:58:39.325743 2912 mem_tracker.cc:140] MemTracker: hard memory limit is 11.777661 GB I1010 20:58:39.325752 2912 mem_tracker.cc:142] MemTracker: soft memory limit is 7.066597 GB I1010 20:58:39.326640 2912 tablet_server_main.cc:54] Initializing tablet server... I1010 20:58:39.326772 2912 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 970615 I1010 20:58:39.327678 2912 server_base.cc:167] Could not load existing FS layout: Not found: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-4/instance: No such file or directory (error 2) I1010 20:58:39.327694 2912 server_base.cc:168] Creating new FS layout I1010 20:58:39.327955 2442 tablet_bootstrap.cc:380] T 00000000000000000000000000000000 P 89d5c63c41ed40f5b968d37d123c09c4: Bootstrap starting. I1010 20:58:39.328114 2442 tablet_bootstrap.cc:542] T 00000000000000000000000000000000 P 89d5c63c41ed40f5b968d37d123c09c4: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:39.328219 2442 tablet_bootstrap.cc:482] T 00000000000000000000000000000000 P 89d5c63c41ed40f5b968d37d123c09c4: No blocks or log segments found. Creating new log. I1010 20:58:39.328341 2442 log.cc:347] Log is configured to *not* fsync() on all Append() calls I1010 20:58:39.328974 2912 fs_manager.cc:354] Generated new instance metadata in path /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-4/instance: uuid: "b8062d66fd284035ae3bc2567bc11e7c" format_stamp: "Formatted at 2016-10-10 20:58:39 on dist-test-slave-dist-test-slave-ymsa" I1010 20:58:39.329673 2912 fs_manager.cc:243] Opened local filesystem: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-4 uuid: "b8062d66fd284035ae3bc2567bc11e7c" format_stamp: "Formatted at 2016-10-10 20:58:39 on dist-test-slave-dist-test-slave-ymsa" I1010 20:58:39.330570 2442 tablet_bootstrap.cc:380] T 00000000000000000000000000000000 P 89d5c63c41ed40f5b968d37d123c09c4: No bootstrap required, opened a new log I1010 20:58:39.330901 2442 raft_consensus.cc:275] T 00000000000000000000000000000000 P 89d5c63c41ed40f5b968d37d123c09c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d5c63c41ed40f5b968d37d123c09c4" member_type: VOTER } I1010 20:58:39.330924 2442 raft_consensus.cc:306] T 00000000000000000000000000000000 P 89d5c63c41ed40f5b968d37d123c09c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:39.330947 2442 raft_consensus.cc:502] T 00000000000000000000000000000000 P 89d5c63c41ed40f5b968d37d123c09c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d5c63c41ed40f5b968d37d123c09c4, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:39.330974 2442 consensus_queue.cc:168] T 00000000000000000000000000000000 P 89d5c63c41ed40f5b968d37d123c09c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:39.330981 2442 raft_consensus.cc:319] T 00000000000000000000000000000000 P 89d5c63c41ed40f5b968d37d123c09c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I1010 20:58:39.330986 2442 raft_consensus.cc:375] T 00000000000000000000000000000000 P 89d5c63c41ed40f5b968d37d123c09c4 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:39.330988 2442 raft_consensus.cc:2031] T 00000000000000000000000000000000 P 89d5c63c41ed40f5b968d37d123c09c4 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:39.331249 2442 raft_consensus.cc:1981] T 00000000000000000000000000000000 P 89d5c63c41ed40f5b968d37d123c09c4 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.633s I1010 20:58:39.331275 2442 raft_consensus.cc:391] T 00000000000000000000000000000000 P 89d5c63c41ed40f5b968d37d123c09c4 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d5c63c41ed40f5b968d37d123c09c4" member_type: VOTER } I1010 20:58:39.331964 2912 tablet_server_main.cc:57] Starting tablet server... I1010 20:58:39.339582 2912 rpc_server.cc:164] RPC server started. Bound to: 127.6.150.4:44055 I1010 20:58:39.339617 2912 webserver.cc:125] Starting webserver on 127.6.150.4:0 I1010 20:58:39.339620 2912 webserver.cc:136] Document root disabled I1010 20:58:39.339706 2912 webserver.cc:220] Webserver started. Bound to: http://127.6.150.4:33555/ I1010 20:58:39.339799 2912 server_base.cc:239] Dumped server information to /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-4/info.pb I1010 20:58:39.340013 2912 tablet_server_main.cc:60] Tablet server successfully started. I1010 20:58:39.340617 3029 heartbeater.cc:292] Connected to a master server at 127.0.0.1:51244 I1010 20:58:39.340633 3029 heartbeater.cc:359] Registering TS with master... I1010 20:58:39.340642 3029 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:39.340785 3029 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:58:39.340797 3029 heartbeater.cc:359] Registering TS with master... I1010 20:58:39.340806 3029 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:39.340965 3029 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:58:39.340978 3029 heartbeater.cc:359] Registering TS with master... I1010 20:58:39.340987 3029 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:39.341138 3029 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 W1010 20:58:39.341161 3029 heartbeater.cc:323] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. I1010 20:58:39.341730 1686 external_mini_cluster.cc:622] Started /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver as pid 2912 I1010 20:58:39.341802 1686 external_mini_cluster.cc:586] Running /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-5 --fs_data_dirs=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-5 --rpc_bind_addresses=127.6.150.5:0 --local_ip_for_outbound_sockets=127.6.150.5 --webserver_interface=127.6.150.5 --webserver_port=0 --tserver_master_addrs=127.0.0.1:51244 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-5 --never_fsync --never_fsync --server_dump_info_path=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-5/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 --unlock_experimental_flags --unlock_unsafe_flags WARNING: Logging before InitGoogleLogging() is written to STDERR W1010 20:58:39.345530 3032 flags.cc:294] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1010 20:58:39.345610 3032 flags.cc:294] Enabled unsafe flag: --never_fsync=true W1010 20:58:39.346034 3032 flags.cc:294] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.150.5 I1010 20:58:39.346117 3032 mem_tracker.cc:140] MemTracker: hard memory limit is 11.777661 GB I1010 20:58:39.346128 3032 mem_tracker.cc:142] MemTracker: soft memory limit is 7.066597 GB I1010 20:58:39.347051 3032 tablet_server_main.cc:54] Initializing tablet server... I1010 20:58:39.347177 3032 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 970615 I1010 20:58:39.347816 3032 server_base.cc:167] Could not load existing FS layout: Not found: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-5/instance: No such file or directory (error 2) I1010 20:58:39.347827 3032 server_base.cc:168] Creating new FS layout I1010 20:58:39.348944 3032 fs_manager.cc:354] Generated new instance metadata in path /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-5/instance: uuid: "33ef4317e50042ba95e517992caa6531" format_stamp: "Formatted at 2016-10-10 20:58:39 on dist-test-slave-dist-test-slave-ymsa" I1010 20:58:39.349601 3032 fs_manager.cc:243] Opened local filesystem: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-5 uuid: "33ef4317e50042ba95e517992caa6531" format_stamp: "Formatted at 2016-10-10 20:58:39 on dist-test-slave-dist-test-slave-ymsa" I1010 20:58:39.351876 3032 tablet_server_main.cc:57] Starting tablet server... I1010 20:58:39.360219 3032 rpc_server.cc:164] RPC server started. Bound to: 127.6.150.5:40764 I1010 20:58:39.360261 3032 webserver.cc:125] Starting webserver on 127.6.150.5:0 I1010 20:58:39.360285 3032 webserver.cc:136] Document root disabled I1010 20:58:39.360455 3032 webserver.cc:220] Webserver started. Bound to: http://127.6.150.5:58212/ I1010 20:58:39.360558 3032 server_base.cc:239] Dumped server information to /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-5/info.pb I1010 20:58:39.360818 3032 tablet_server_main.cc:60] Tablet server successfully started. I1010 20:58:39.361472 3146 heartbeater.cc:292] Connected to a master server at 127.0.0.1:51244 I1010 20:58:39.361490 3146 heartbeater.cc:359] Registering TS with master... I1010 20:58:39.361500 3146 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:39.361654 3146 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:58:39.361668 3146 heartbeater.cc:359] Registering TS with master... I1010 20:58:39.361678 3146 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:39.361804 3146 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:58:39.361817 3146 heartbeater.cc:359] Registering TS with master... I1010 20:58:39.361826 3146 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:39.361955 3146 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 W1010 20:58:39.361968 3146 heartbeater.cc:323] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. I1010 20:58:39.362303 1686 external_mini_cluster.cc:622] Started /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver as pid 3032 I1010 20:58:39.362350 1686 external_mini_cluster.cc:586] Running /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-6 --fs_data_dirs=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-6 --rpc_bind_addresses=127.6.150.6:0 --local_ip_for_outbound_sockets=127.6.150.6 --webserver_interface=127.6.150.6 --webserver_port=0 --tserver_master_addrs=127.0.0.1:51244 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-6 --never_fsync --never_fsync --server_dump_info_path=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-6/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 --unlock_experimental_flags --unlock_unsafe_flags WARNING: Logging before InitGoogleLogging() is written to STDERR W1010 20:58:39.366247 3149 flags.cc:294] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1010 20:58:39.366302 3149 flags.cc:294] Enabled unsafe flag: --never_fsync=true W1010 20:58:39.366693 3149 flags.cc:294] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.150.6 I1010 20:58:39.366773 3149 mem_tracker.cc:140] MemTracker: hard memory limit is 11.777661 GB I1010 20:58:39.366786 3149 mem_tracker.cc:142] MemTracker: soft memory limit is 7.066597 GB I1010 20:58:39.367662 3149 tablet_server_main.cc:54] Initializing tablet server... I1010 20:58:39.367794 3149 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 970615 I1010 20:58:39.368439 3149 server_base.cc:167] Could not load existing FS layout: Not found: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-6/instance: No such file or directory (error 2) I1010 20:58:39.368449 3149 server_base.cc:168] Creating new FS layout I1010 20:58:39.369345 3149 fs_manager.cc:354] Generated new instance metadata in path /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-6/instance: uuid: "1380a27d6b824575a3a5e139bee139f4" format_stamp: "Formatted at 2016-10-10 20:58:39 on dist-test-slave-dist-test-slave-ymsa" I1010 20:58:39.370019 3149 fs_manager.cc:243] Opened local filesystem: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-6 uuid: "1380a27d6b824575a3a5e139bee139f4" format_stamp: "Formatted at 2016-10-10 20:58:39 on dist-test-slave-dist-test-slave-ymsa" I1010 20:58:39.372426 3149 tablet_server_main.cc:57] Starting tablet server... I1010 20:58:39.380468 3149 rpc_server.cc:164] RPC server started. Bound to: 127.6.150.6:46774 I1010 20:58:39.380518 3149 webserver.cc:125] Starting webserver on 127.6.150.6:0 I1010 20:58:39.380522 3149 webserver.cc:136] Document root disabled I1010 20:58:39.380626 3149 webserver.cc:220] Webserver started. Bound to: http://127.6.150.6:36048/ I1010 20:58:39.380718 3149 server_base.cc:239] Dumped server information to /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-6/info.pb I1010 20:58:39.380966 3149 tablet_server_main.cc:60] Tablet server successfully started. I1010 20:58:39.382005 3263 heartbeater.cc:292] Connected to a master server at 127.0.0.1:51244 I1010 20:58:39.382024 3263 heartbeater.cc:359] Registering TS with master... I1010 20:58:39.382033 3263 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:39.382210 3263 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:58:39.382223 3263 heartbeater.cc:359] Registering TS with master... I1010 20:58:39.382233 3263 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:39.382380 3263 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:58:39.382397 3263 heartbeater.cc:359] Registering TS with master... I1010 20:58:39.382407 3263 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:39.382536 3263 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 W1010 20:58:39.382549 3263 heartbeater.cc:323] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. I1010 20:58:39.382794 1686 external_mini_cluster.cc:622] Started /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver as pid 3149 I1010 20:58:39.382838 1686 external_mini_cluster.cc:586] Running /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-7 --fs_data_dirs=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-7 --rpc_bind_addresses=127.6.150.7:0 --local_ip_for_outbound_sockets=127.6.150.7 --webserver_interface=127.6.150.7 --webserver_port=0 --tserver_master_addrs=127.0.0.1:51244 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-7 --never_fsync --never_fsync --server_dump_info_path=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-7/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 --unlock_experimental_flags --unlock_unsafe_flags WARNING: Logging before InitGoogleLogging() is written to STDERR W1010 20:58:39.386667 3266 flags.cc:294] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1010 20:58:39.386764 3266 flags.cc:294] Enabled unsafe flag: --never_fsync=true W1010 20:58:39.387116 3266 flags.cc:294] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.150.7 I1010 20:58:39.387177 3266 mem_tracker.cc:140] MemTracker: hard memory limit is 11.777661 GB I1010 20:58:39.387186 3266 mem_tracker.cc:142] MemTracker: soft memory limit is 7.066597 GB I1010 20:58:39.388057 3266 tablet_server_main.cc:54] Initializing tablet server... I1010 20:58:39.388162 3266 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 970615 I1010 20:58:39.388746 3266 server_base.cc:167] Could not load existing FS layout: Not found: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-7/instance: No such file or directory (error 2) I1010 20:58:39.388754 3266 server_base.cc:168] Creating new FS layout I1010 20:58:39.389714 3266 fs_manager.cc:354] Generated new instance metadata in path /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-7/instance: uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" format_stamp: "Formatted at 2016-10-10 20:58:39 on dist-test-slave-dist-test-slave-ymsa" I1010 20:58:39.390319 3266 fs_manager.cc:243] Opened local filesystem: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-7 uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" format_stamp: "Formatted at 2016-10-10 20:58:39 on dist-test-slave-dist-test-slave-ymsa" I1010 20:58:39.392910 3266 tablet_server_main.cc:57] Starting tablet server... I1010 20:58:39.401244 3266 rpc_server.cc:164] RPC server started. Bound to: 127.6.150.7:45436 I1010 20:58:39.401290 3266 webserver.cc:125] Starting webserver on 127.6.150.7:0 I1010 20:58:39.401316 3266 webserver.cc:136] Document root disabled I1010 20:58:39.401458 3266 webserver.cc:220] Webserver started. Bound to: http://127.6.150.7:48624/ I1010 20:58:39.401556 3266 server_base.cc:239] Dumped server information to /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-7/info.pb I1010 20:58:39.401754 3266 tablet_server_main.cc:60] Tablet server successfully started. I1010 20:58:39.402462 3380 heartbeater.cc:292] Connected to a master server at 127.0.0.1:51244 I1010 20:58:39.402490 3380 heartbeater.cc:359] Registering TS with master... I1010 20:58:39.402501 3380 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:39.402720 3380 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:58:39.402734 3380 heartbeater.cc:359] Registering TS with master... I1010 20:58:39.402760 3380 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:39.402979 3380 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:58:39.402993 3380 heartbeater.cc:359] Registering TS with master... I1010 20:58:39.403003 3380 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:39.403154 3380 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 W1010 20:58:39.403167 3380 heartbeater.cc:323] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. I1010 20:58:39.403285 1686 external_mini_cluster.cc:622] Started /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver as pid 3266 I1010 20:58:39.403331 1686 external_mini_cluster.cc:586] Running /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-8 --fs_data_dirs=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-8 --rpc_bind_addresses=127.6.150.8:0 --local_ip_for_outbound_sockets=127.6.150.8 --webserver_interface=127.6.150.8 --webserver_port=0 --tserver_master_addrs=127.0.0.1:51244 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-8 --never_fsync --never_fsync --server_dump_info_path=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-8/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 --unlock_experimental_flags --unlock_unsafe_flags WARNING: Logging before InitGoogleLogging() is written to STDERR W1010 20:58:39.407232 3383 flags.cc:294] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1010 20:58:39.407287 3383 flags.cc:294] Enabled unsafe flag: --never_fsync=true W1010 20:58:39.407658 3383 flags.cc:294] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.150.8 I1010 20:58:39.407723 3383 mem_tracker.cc:140] MemTracker: hard memory limit is 11.777661 GB I1010 20:58:39.407732 3383 mem_tracker.cc:142] MemTracker: soft memory limit is 7.066597 GB I1010 20:58:39.408615 3383 tablet_server_main.cc:54] Initializing tablet server... I1010 20:58:39.408721 3383 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 970615 I1010 20:58:39.409317 3383 server_base.cc:167] Could not load existing FS layout: Not found: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-8/instance: No such file or directory (error 2) I1010 20:58:39.409327 3383 server_base.cc:168] Creating new FS layout I1010 20:58:39.410223 3383 fs_manager.cc:354] Generated new instance metadata in path /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-8/instance: uuid: "42e94fc3adbb419591836d222d946978" format_stamp: "Formatted at 2016-10-10 20:58:39 on dist-test-slave-dist-test-slave-ymsa" I1010 20:58:39.410790 3383 fs_manager.cc:243] Opened local filesystem: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-8 uuid: "42e94fc3adbb419591836d222d946978" format_stamp: "Formatted at 2016-10-10 20:58:39 on dist-test-slave-dist-test-slave-ymsa" I1010 20:58:39.412731 3383 tablet_server_main.cc:57] Starting tablet server... I1010 20:58:39.420915 3383 rpc_server.cc:164] RPC server started. Bound to: 127.6.150.8:35299 I1010 20:58:39.420966 3383 webserver.cc:125] Starting webserver on 127.6.150.8:0 I1010 20:58:39.420972 3383 webserver.cc:136] Document root disabled I1010 20:58:39.421064 3383 webserver.cc:220] Webserver started. Bound to: http://127.6.150.8:46395/ I1010 20:58:39.421164 3383 server_base.cc:239] Dumped server information to /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-8/info.pb I1010 20:58:39.421358 3383 tablet_server_main.cc:60] Tablet server successfully started. I1010 20:58:39.422123 3497 heartbeater.cc:292] Connected to a master server at 127.0.0.1:51244 I1010 20:58:39.422140 3497 heartbeater.cc:359] Registering TS with master... I1010 20:58:39.422154 3497 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:39.422358 3497 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:58:39.422370 3497 heartbeater.cc:359] Registering TS with master... I1010 20:58:39.422391 3497 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:39.422582 3497 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:58:39.422601 3497 heartbeater.cc:359] Registering TS with master... I1010 20:58:39.422613 3497 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:39.422780 3497 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 W1010 20:58:39.422791 3497 heartbeater.cc:323] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. I1010 20:58:39.423805 1686 external_mini_cluster.cc:622] Started /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver as pid 3383 I1010 20:58:39.423854 1686 external_mini_cluster.cc:586] Running /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-9 --fs_data_dirs=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-9 --rpc_bind_addresses=127.6.150.9:0 --local_ip_for_outbound_sockets=127.6.150.9 --webserver_interface=127.6.150.9 --webserver_port=0 --tserver_master_addrs=127.0.0.1:51244 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-9 --never_fsync --never_fsync --server_dump_info_path=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-9/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 --unlock_experimental_flags --unlock_unsafe_flags WARNING: Logging before InitGoogleLogging() is written to STDERR W1010 20:58:39.427896 3500 flags.cc:294] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1010 20:58:39.427959 3500 flags.cc:294] Enabled unsafe flag: --never_fsync=true W1010 20:58:39.428321 3500 flags.cc:294] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.150.9 I1010 20:58:39.428382 3500 mem_tracker.cc:140] MemTracker: hard memory limit is 11.777661 GB I1010 20:58:39.428390 3500 mem_tracker.cc:142] MemTracker: soft memory limit is 7.066597 GB I1010 20:58:39.429251 3500 tablet_server_main.cc:54] Initializing tablet server... I1010 20:58:39.429358 3500 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 970615 I1010 20:58:39.429970 3500 server_base.cc:167] Could not load existing FS layout: Not found: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-9/instance: No such file or directory (error 2) I1010 20:58:39.429981 3500 server_base.cc:168] Creating new FS layout I1010 20:58:39.430881 3500 fs_manager.cc:354] Generated new instance metadata in path /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-9/instance: uuid: "580cbaf7092444dfbc6fb344da88d7b6" format_stamp: "Formatted at 2016-10-10 20:58:39 on dist-test-slave-dist-test-slave-ymsa" I1010 20:58:39.431466 3500 fs_manager.cc:243] Opened local filesystem: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-9 uuid: "580cbaf7092444dfbc6fb344da88d7b6" format_stamp: "Formatted at 2016-10-10 20:58:39 on dist-test-slave-dist-test-slave-ymsa" I1010 20:58:39.433553 3500 tablet_server_main.cc:57] Starting tablet server... I1010 20:58:39.441823 3500 rpc_server.cc:164] RPC server started. Bound to: 127.6.150.9:51743 I1010 20:58:39.441857 3500 webserver.cc:125] Starting webserver on 127.6.150.9:0 I1010 20:58:39.441860 3500 webserver.cc:136] Document root disabled I1010 20:58:39.441956 3500 webserver.cc:220] Webserver started. Bound to: http://127.6.150.9:43279/ I1010 20:58:39.442045 3500 server_base.cc:239] Dumped server information to /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686/minicluster-data/ts-9/info.pb I1010 20:58:39.442255 3500 tablet_server_main.cc:60] Tablet server successfully started. I1010 20:58:39.442805 3614 heartbeater.cc:292] Connected to a master server at 127.0.0.1:51244 I1010 20:58:39.442819 3614 heartbeater.cc:359] Registering TS with master... I1010 20:58:39.442831 3614 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:39.442991 3614 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:58:39.443016 3614 heartbeater.cc:359] Registering TS with master... I1010 20:58:39.443037 3614 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:39.443229 3614 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:58:39.443241 3614 heartbeater.cc:359] Registering TS with master... I1010 20:58:39.443253 3614 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:39.443392 3614 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 W1010 20:58:39.443403 3614 heartbeater.cc:323] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. I1010 20:58:39.444511 1686 external_mini_cluster.cc:622] Started /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver as pid 3500 I1010 20:58:39.892796 2557 heartbeater.cc:292] Connected to a master server at 127.0.0.1:51244 I1010 20:58:39.892814 2557 heartbeater.cc:359] Registering TS with master... I1010 20:58:39.892824 2557 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:39.893016 2557 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:58:40.261828 2675 heartbeater.cc:292] Connected to a master server at 127.0.0.1:51244 I1010 20:58:40.261852 2675 heartbeater.cc:359] Registering TS with master... I1010 20:58:40.261865 2675 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:40.262122 2675 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:58:40.292847 2792 heartbeater.cc:292] Connected to a master server at 127.0.0.1:51244 I1010 20:58:40.292863 2792 heartbeater.cc:359] Registering TS with master... I1010 20:58:40.292873 2792 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:40.293048 2792 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:58:40.321250 2909 heartbeater.cc:292] Connected to a master server at 127.0.0.1:51244 I1010 20:58:40.321269 2909 heartbeater.cc:359] Registering TS with master... I1010 20:58:40.321281 2909 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:40.321424 2909 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:58:40.341548 3029 heartbeater.cc:292] Connected to a master server at 127.0.0.1:51244 I1010 20:58:40.341565 3029 heartbeater.cc:359] Registering TS with master... I1010 20:58:40.341578 3029 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:40.341775 3029 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:58:40.362326 3146 heartbeater.cc:292] Connected to a master server at 127.0.0.1:51244 I1010 20:58:40.362354 3146 heartbeater.cc:359] Registering TS with master... I1010 20:58:40.362375 3146 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:40.362578 3146 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:58:40.382848 3263 heartbeater.cc:292] Connected to a master server at 127.0.0.1:51244 I1010 20:58:40.382864 3263 heartbeater.cc:359] Registering TS with master... I1010 20:58:40.382874 3263 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:40.383018 3263 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:58:40.403515 3380 heartbeater.cc:292] Connected to a master server at 127.0.0.1:51244 I1010 20:58:40.403535 3380 heartbeater.cc:359] Registering TS with master... I1010 20:58:40.403549 3380 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:40.403769 3380 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:58:40.423112 3497 heartbeater.cc:292] Connected to a master server at 127.0.0.1:51244 I1010 20:58:40.423131 3497 heartbeater.cc:359] Registering TS with master... I1010 20:58:40.423143 3497 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:40.423358 3497 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:58:40.443737 3614 heartbeater.cc:292] Connected to a master server at 127.0.0.1:51244 I1010 20:58:40.443750 3614 heartbeater.cc:359] Registering TS with master... I1010 20:58:40.443761 3614 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:40.443958 3614 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:58:40.893404 2557 heartbeater.cc:292] Connected to a master server at 127.0.0.1:51244 I1010 20:58:40.893424 2557 heartbeater.cc:359] Registering TS with master... I1010 20:58:40.893438 2557 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... W1010 20:58:40.893646 2557 heartbeater.cc:453] Failed to heartbeat to 127.0.0.1:51244: Service unavailable: Catalog manager is not initialized. State: 1 W1010 20:58:40.905724 2442 consensus_meta.cc:188] T 00000000000000000000000000000000 P 89d5c63c41ed40f5b968d37d123c09c4: Time spent flushing consensus metadata: real 1.574s user 0.000s sys 0.000s I1010 20:58:40.905747 2442 leader_election.cc:248] T 00000000000000000000000000000000 P 89d5c63c41ed40f5b968d37d123c09c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I1010 20:58:40.906030 2442 sys_catalog.cc:257] T 00000000000000000000000000000000 P 89d5c63c41ed40f5b968d37d123c09c4 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletPeer. Latest consensus state: current_term: 1 config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d5c63c41ed40f5b968d37d123c09c4" member_type: VOTER } } I1010 20:58:40.906033 3618 raft_consensus.cc:1981] T 00000000000000000000000000000000 P 89d5c63c41ed40f5b968d37d123c09c4 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.594s I1010 20:58:40.906050 2442 sys_catalog.cc:260] T 00000000000000000000000000000000 P 89d5c63c41ed40f5b968d37d123c09c4 [sys.catalog]: This master's current role is: FOLLOWER I1010 20:58:40.906090 3618 raft_consensus.cc:1861] T 00000000000000000000000000000000 P 89d5c63c41ed40f5b968d37d123c09c4 [term 1 FOLLOWER]: Leader election won for term 1 I1010 20:58:40.906213 3618 raft_consensus.cc:467] T 00000000000000000000000000000000 P 89d5c63c41ed40f5b968d37d123c09c4 [term 1 LEADER]: Becoming Leader. State: Replica: 89d5c63c41ed40f5b968d37d123c09c4, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:40.906253 3618 consensus_queue.cc:151] T 00000000000000000000000000000000 P 89d5c63c41ed40f5b968d37d123c09c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 1, Majority size: 1, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d5c63c41ed40f5b968d37d123c09c4" member_type: VOTER } I1010 20:58:40.907760 3620 sys_catalog.cc:257] T 00000000000000000000000000000000 P 89d5c63c41ed40f5b968d37d123c09c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89d5c63c41ed40f5b968d37d123c09c4. Latest consensus state: current_term: 1 leader_uuid: "89d5c63c41ed40f5b968d37d123c09c4" config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d5c63c41ed40f5b968d37d123c09c4" member_type: VOTER } } I1010 20:58:40.907774 3620 sys_catalog.cc:260] T 00000000000000000000000000000000 P 89d5c63c41ed40f5b968d37d123c09c4 [sys.catalog]: This master's current role is: LEADER I1010 20:58:40.907847 2442 sys_catalog.cc:335] T 00000000000000000000000000000000 P 89d5c63c41ed40f5b968d37d123c09c4 [sys.catalog]: configured and running, proceeding with master startup. I1010 20:58:40.907969 3619 sys_catalog.cc:257] T 00000000000000000000000000000000 P 89d5c63c41ed40f5b968d37d123c09c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89d5c63c41ed40f5b968d37d123c09c4" config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d5c63c41ed40f5b968d37d123c09c4" member_type: VOTER } } I1010 20:58:40.907980 3619 sys_catalog.cc:260] T 00000000000000000000000000000000 P 89d5c63c41ed40f5b968d37d123c09c4 [sys.catalog]: This master's current role is: LEADER W1010 20:58:40.908067 3622 catalog_manager.cc:365] Catalog manager background task thread going to sleep: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:58:40.908102 2388 master_main.cc:64] Master server successfully started. I1010 20:58:40.908175 3621 catalog_manager.cc:660] Loading table and tablet metadata into memory... I1010 20:58:41.262645 2675 heartbeater.cc:292] Connected to a master server at 127.0.0.1:51244 I1010 20:58:41.262697 2675 heartbeater.cc:359] Registering TS with master... I1010 20:58:41.262715 2675 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... I1010 20:58:41.262819 2408 ts_manager.cc:76] Registered new tserver with Master: 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963) I1010 20:58:41.293474 2792 heartbeater.cc:292] Connected to a master server at 127.0.0.1:51244 I1010 20:58:41.293494 2792 heartbeater.cc:359] Registering TS with master... I1010 20:58:41.293507 2792 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... I1010 20:58:41.293601 2408 ts_manager.cc:76] Registered new tserver with Master: f6d15ccd64ec49be8396ed78e2a88377 (127.6.150.2:33554) I1010 20:58:41.321826 2909 heartbeater.cc:292] Connected to a master server at 127.0.0.1:51244 I1010 20:58:41.321851 2909 heartbeater.cc:359] Registering TS with master... I1010 20:58:41.321861 2909 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... I1010 20:58:41.321997 2408 ts_manager.cc:76] Registered new tserver with Master: 8de6cfa864d64c58bf4eafb689517c93 (127.6.150.3:40117) I1010 20:58:41.342193 3029 heartbeater.cc:292] Connected to a master server at 127.0.0.1:51244 I1010 20:58:41.342209 3029 heartbeater.cc:359] Registering TS with master... I1010 20:58:41.342221 3029 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... I1010 20:58:41.342335 2409 ts_manager.cc:76] Registered new tserver with Master: b8062d66fd284035ae3bc2567bc11e7c (127.6.150.4:44055) I1010 20:58:41.362907 3146 heartbeater.cc:292] Connected to a master server at 127.0.0.1:51244 I1010 20:58:41.362922 3146 heartbeater.cc:359] Registering TS with master... I1010 20:58:41.362941 3146 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... I1010 20:58:41.363030 2409 ts_manager.cc:76] Registered new tserver with Master: 33ef4317e50042ba95e517992caa6531 (127.6.150.5:40764) I1010 20:58:41.383316 3263 heartbeater.cc:292] Connected to a master server at 127.0.0.1:51244 I1010 20:58:41.383333 3263 heartbeater.cc:359] Registering TS with master... I1010 20:58:41.383345 3263 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... I1010 20:58:41.383424 2408 ts_manager.cc:76] Registered new tserver with Master: 1380a27d6b824575a3a5e139bee139f4 (127.6.150.6:46774) I1010 20:58:41.404091 3380 heartbeater.cc:292] Connected to a master server at 127.0.0.1:51244 I1010 20:58:41.404109 3380 heartbeater.cc:359] Registering TS with master... I1010 20:58:41.404117 3380 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... I1010 20:58:41.404207 2409 ts_manager.cc:76] Registered new tserver with Master: 7f623b00f5ae4b9c8e8940692d5dfd44 (127.6.150.7:45436) I1010 20:58:41.423683 3497 heartbeater.cc:292] Connected to a master server at 127.0.0.1:51244 I1010 20:58:41.423698 3497 heartbeater.cc:359] Registering TS with master... I1010 20:58:41.423708 3497 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... I1010 20:58:41.423784 2409 ts_manager.cc:76] Registered new tserver with Master: 42e94fc3adbb419591836d222d946978 (127.6.150.8:35299) I1010 20:58:41.444393 3614 heartbeater.cc:292] Connected to a master server at 127.0.0.1:51244 I1010 20:58:41.444422 3614 heartbeater.cc:359] Registering TS with master... I1010 20:58:41.444435 3614 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... I1010 20:58:41.444581 2409 ts_manager.cc:76] Registered new tserver with Master: 580cbaf7092444dfbc6fb344da88d7b6 (127.6.150.9:51743) I1010 20:58:41.893993 2557 heartbeater.cc:292] Connected to a master server at 127.0.0.1:51244 I1010 20:58:41.894019 2557 heartbeater.cc:359] Registering TS with master... I1010 20:58:41.894031 2557 heartbeater.cc:373] Master 127.0.0.1:51244 requested a full tablet report, sending... I1010 20:58:41.894168 2409 ts_manager.cc:76] Registered new tserver with Master: 84bde34331b94c4ea003e48f9317f792 (127.6.150.0:40113) I1010 20:58:41.894353 1686 external_mini_cluster.cc:341] 10 TS(s) registered with all masters I1010 20:58:41.897218 2408 catalog_manager.cc:789] CreateTable from {real_user=slave, eff_user=} at 127.0.0.1:34962: name: "test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 20 seed: 0 } range_schema { columns { name: "key" } } } I1010 20:58:41.899900 2846 tablet_service.cc:619] Processing CreateTablet for tablet 872042177fee42c9b6c380a1b657755e (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (0), range: [, ) I1010 20:58:41.900892 2724 tablet_service.cc:619] Processing CreateTablet for tablet 4c179cfa32ee4128ad57d34140e5320f (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (15), range: [, ) I1010 20:58:41.900900 2728 tablet_service.cc:619] Processing CreateTablet for tablet 05a5666441ec49d792222a3e8674edf7 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (3), range: [, ) I1010 20:58:41.901072 2727 tablet_service.cc:619] Processing CreateTablet for tablet efe8fb09a1234c9fb1dc99948737a894 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (7), range: [, ) I1010 20:58:41.901105 2846 ts_tablet_manager.cc:766] Registered tablet 872042177fee42c9b6c380a1b657755e I1010 20:58:41.901147 2726 tablet_service.cc:619] Processing CreateTablet for tablet 90fa7ae3643e4575a6be1c4936b6cc11 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (9), range: [, ) I1010 20:58:41.901258 2725 tablet_service.cc:619] Processing CreateTablet for tablet f689db1eddb54b1f85b7df38d14534b4 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (14), range: [, ) I1010 20:58:41.901420 2845 tablet_service.cc:619] Processing CreateTablet for tablet 949b26dfeadc4de38868bf4ed994d184 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (4), range: [, ) I1010 20:58:41.901784 2729 tablet_service.cc:619] Processing CreateTablet for tablet 872042177fee42c9b6c380a1b657755e (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (0), range: [, ) I1010 20:58:41.901844 3083 tablet_service.cc:619] Processing CreateTablet for tablet 7d680e6840854bca9daef77d506408e9 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (1), range: [, ) I1010 20:58:41.901896 2845 ts_tablet_manager.cc:766] Registered tablet 949b26dfeadc4de38868bf4ed994d184 I1010 20:58:41.901988 2844 tablet_service.cc:619] Processing CreateTablet for tablet 4aa163baddb74c4998157964dd55b1f4 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (8), range: [, ) I1010 20:58:41.902611 2844 ts_tablet_manager.cc:766] Registered tablet 4aa163baddb74c4998157964dd55b1f4 I1010 20:58:41.902648 2727 ts_tablet_manager.cc:766] Registered tablet efe8fb09a1234c9fb1dc99948737a894 I1010 20:58:41.902658 2843 tablet_service.cc:619] Processing CreateTablet for tablet 981834f5e88f4585b6420fca7ddcb289 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (10), range: [, ) I1010 20:58:41.902724 2842 tablet_service.cc:619] Processing CreateTablet for tablet 19f492d94e864f1ebd1c70810b0470f7 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (12), range: [, ) I1010 20:58:41.903226 3083 ts_tablet_manager.cc:766] Registered tablet 7d680e6840854bca9daef77d506408e9 I1010 20:58:41.903286 2842 ts_tablet_manager.cc:766] Registered tablet 19f492d94e864f1ebd1c70810b0470f7 I1010 20:58:41.903313 3082 tablet_service.cc:619] Processing CreateTablet for tablet 05a5666441ec49d792222a3e8674edf7 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (3), range: [, ) I1010 20:58:41.903355 2841 tablet_service.cc:619] Processing CreateTablet for tablet de9d164718e74d3db902ea603ab32407 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (16), range: [, ) I1010 20:58:41.903692 3082 ts_tablet_manager.cc:766] Registered tablet 05a5666441ec49d792222a3e8674edf7 I1010 20:58:41.903726 3081 tablet_service.cc:619] Processing CreateTablet for tablet 8da5904fd56d441ea29e61eb9d2b0b2f (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (11), range: [, ) I1010 20:58:41.903760 2841 ts_tablet_manager.cc:766] Registered tablet de9d164718e74d3db902ea603ab32407 I1010 20:58:41.903803 2840 tablet_service.cc:619] Processing CreateTablet for tablet 7836528275cb45aab827361611bb3bbe (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (18), range: [, ) I1010 20:58:41.904055 3081 ts_tablet_manager.cc:766] Registered tablet 8da5904fd56d441ea29e61eb9d2b0b2f I1010 20:58:41.904093 3080 tablet_service.cc:619] Processing CreateTablet for tablet 19f492d94e864f1ebd1c70810b0470f7 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (12), range: [, ) I1010 20:58:41.904193 2840 ts_tablet_manager.cc:766] Registered tablet 7836528275cb45aab827361611bb3bbe I1010 20:58:41.904361 3080 ts_tablet_manager.cc:766] Registered tablet 19f492d94e864f1ebd1c70810b0470f7 I1010 20:58:41.904405 3079 tablet_service.cc:619] Processing CreateTablet for tablet 4c179cfa32ee4128ad57d34140e5320f (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (15), range: [, ) I1010 20:58:41.904624 3434 tablet_service.cc:619] Processing CreateTablet for tablet 872042177fee42c9b6c380a1b657755e (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (0), range: [, ) I1010 20:58:41.904697 3433 tablet_service.cc:619] Processing CreateTablet for tablet f2289fb138ea4a5ab6c02803c4e9aa51 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (6), range: [, ) I1010 20:58:41.904805 2729 ts_tablet_manager.cc:766] Registered tablet 872042177fee42c9b6c380a1b657755e I1010 20:58:41.904858 3079 ts_tablet_manager.cc:766] Registered tablet 4c179cfa32ee4128ad57d34140e5320f I1010 20:58:41.904902 3078 tablet_service.cc:619] Processing CreateTablet for tablet ed24ca4817744964991178277119b367 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (17), range: [, ) I1010 20:58:41.904978 2726 ts_tablet_manager.cc:766] Registered tablet 90fa7ae3643e4575a6be1c4936b6cc11 I1010 20:58:41.905079 2494 tablet_service.cc:619] Processing CreateTablet for tablet 7d680e6840854bca9daef77d506408e9 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (1), range: [, ) I1010 20:58:41.905164 2728 ts_tablet_manager.cc:766] Registered tablet 05a5666441ec49d792222a3e8674edf7 I1010 20:58:41.905179 2493 tablet_service.cc:619] Processing CreateTablet for tablet 95e7c143b0524a9b9ade2cfcc081e679 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (5), range: [, ) I1010 20:58:41.905320 3078 ts_tablet_manager.cc:766] Registered tablet ed24ca4817744964991178277119b367 I1010 20:58:41.905622 3433 ts_tablet_manager.cc:766] Registered tablet f2289fb138ea4a5ab6c02803c4e9aa51 I1010 20:58:41.905696 3432 tablet_service.cc:619] Processing CreateTablet for tablet 90fa7ae3643e4575a6be1c4936b6cc11 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (9), range: [, ) I1010 20:58:41.905807 3200 tablet_service.cc:619] Processing CreateTablet for tablet 7d680e6840854bca9daef77d506408e9 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (1), range: [, ) I1010 20:58:41.905864 3317 tablet_service.cc:619] Processing CreateTablet for tablet 17a4ac4e4e164cebb4d88f59511cf926 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (2), range: [, ) I1010 20:58:41.906121 3432 ts_tablet_manager.cc:766] Registered tablet 90fa7ae3643e4575a6be1c4936b6cc11 I1010 20:58:41.906131 2492 tablet_service.cc:619] Processing CreateTablet for tablet 4aa163baddb74c4998157964dd55b1f4 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (8), range: [, ) I1010 20:58:41.906165 3431 tablet_service.cc:619] Processing CreateTablet for tablet 84fcfff9fdef4acf8c2b47f36330567f (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (13), range: [, ) I1010 20:58:41.906563 2492 ts_tablet_manager.cc:766] Registered tablet 4aa163baddb74c4998157964dd55b1f4 I1010 20:58:41.906620 3431 ts_tablet_manager.cc:766] Registered tablet 84fcfff9fdef4acf8c2b47f36330567f I1010 20:58:41.906631 3316 tablet_service.cc:619] Processing CreateTablet for tablet 95e7c143b0524a9b9ade2cfcc081e679 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (5), range: [, ) I1010 20:58:41.906636 2491 tablet_service.cc:619] Processing CreateTablet for tablet 8da5904fd56d441ea29e61eb9d2b0b2f (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (11), range: [, ) I1010 20:58:41.906666 3430 tablet_service.cc:619] Processing CreateTablet for tablet f689db1eddb54b1f85b7df38d14534b4 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (14), range: [, ) I1010 20:58:41.906720 3315 tablet_service.cc:619] Processing CreateTablet for tablet f2289fb138ea4a5ab6c02803c4e9aa51 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (6), range: [, ) I1010 20:58:41.906744 2490 tablet_service.cc:619] Processing CreateTablet for tablet 19f492d94e864f1ebd1c70810b0470f7 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (12), range: [, ) I1010 20:58:41.906841 3200 ts_tablet_manager.cc:766] Registered tablet 7d680e6840854bca9daef77d506408e9 I1010 20:58:41.906843 3314 tablet_service.cc:619] Processing CreateTablet for tablet 4c179cfa32ee4128ad57d34140e5320f (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (15), range: [, ) I1010 20:58:41.906909 3199 tablet_service.cc:619] Processing CreateTablet for tablet 949b26dfeadc4de38868bf4ed994d184 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (4), range: [, ) I1010 20:58:41.907044 3429 tablet_service.cc:619] Processing CreateTablet for tablet 7836528275cb45aab827361611bb3bbe (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (18), range: [, ) I1010 20:58:41.907107 3313 tablet_service.cc:619] Processing CreateTablet for tablet de9d164718e74d3db902ea603ab32407 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (16), range: [, ) I1010 20:58:41.907207 2489 tablet_service.cc:619] Processing CreateTablet for tablet 970154c84d6849cc83af1d9e5ad6980d (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (19), range: [, ) I1010 20:58:41.907322 3199 ts_tablet_manager.cc:766] Registered tablet 949b26dfeadc4de38868bf4ed994d184 I1010 20:58:41.907367 3198 tablet_service.cc:619] Processing CreateTablet for tablet 90fa7ae3643e4575a6be1c4936b6cc11 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (9), range: [, ) I1010 20:58:41.907433 3429 ts_tablet_manager.cc:766] Registered tablet 7836528275cb45aab827361611bb3bbe I1010 20:58:41.907604 2489 ts_tablet_manager.cc:766] Registered tablet 970154c84d6849cc83af1d9e5ad6980d I1010 20:58:41.907666 3313 ts_tablet_manager.cc:766] Registered tablet de9d164718e74d3db902ea603ab32407 I1010 20:58:41.907733 3312 tablet_service.cc:619] Processing CreateTablet for tablet ed24ca4817744964991178277119b367 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (17), range: [, ) I1010 20:58:41.907786 3197 tablet_service.cc:619] Processing CreateTablet for tablet 981834f5e88f4585b6420fca7ddcb289 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (10), range: [, ) I1010 20:58:41.908032 2491 ts_tablet_manager.cc:766] Registered tablet 8da5904fd56d441ea29e61eb9d2b0b2f I1010 20:58:41.908046 3648 ts_tablet_manager.cc:632] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377: Bootstrapping tablet I1010 20:58:41.908115 2725 ts_tablet_manager.cc:766] Registered tablet f689db1eddb54b1f85b7df38d14534b4 I1010 20:58:41.908166 3197 ts_tablet_manager.cc:766] Registered tablet 981834f5e88f4585b6420fca7ddcb289 I1010 20:58:41.908211 3196 tablet_service.cc:619] Processing CreateTablet for tablet f689db1eddb54b1f85b7df38d14534b4 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (14), range: [, ) I1010 20:58:41.908318 3648 tablet_bootstrap.cc:380] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377: Bootstrap starting. I1010 20:58:41.908453 2843 ts_tablet_manager.cc:766] Registered tablet 981834f5e88f4585b6420fca7ddcb289 I1010 20:58:41.908792 2966 tablet_service.cc:619] Processing CreateTablet for tablet 949b26dfeadc4de38868bf4ed994d184 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (4), range: [, ) I1010 20:58:41.909868 3434 ts_tablet_manager.cc:766] Registered tablet 872042177fee42c9b6c380a1b657755e I1010 20:58:41.910064 3316 ts_tablet_manager.cc:766] Registered tablet 95e7c143b0524a9b9ade2cfcc081e679 I1010 20:58:41.910070 2965 tablet_service.cc:619] Processing CreateTablet for tablet f2289fb138ea4a5ab6c02803c4e9aa51 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (6), range: [, ) I1010 20:58:41.910228 3312 ts_tablet_manager.cc:766] Registered tablet ed24ca4817744964991178277119b367 I1010 20:58:41.910364 2494 ts_tablet_manager.cc:766] Registered tablet 7d680e6840854bca9daef77d506408e9 I1010 20:58:41.910415 2493 ts_tablet_manager.cc:766] Registered tablet 95e7c143b0524a9b9ade2cfcc081e679 I1010 20:58:41.910477 3551 tablet_service.cc:619] Processing CreateTablet for tablet 17a4ac4e4e164cebb4d88f59511cf926 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (2), range: [, ) I1010 20:58:41.910579 3317 ts_tablet_manager.cc:766] Registered tablet 17a4ac4e4e164cebb4d88f59511cf926 I1010 20:58:41.910625 2965 ts_tablet_manager.cc:766] Registered tablet f2289fb138ea4a5ab6c02803c4e9aa51 I1010 20:58:41.910717 2964 tablet_service.cc:619] Processing CreateTablet for tablet efe8fb09a1234c9fb1dc99948737a894 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (7), range: [, ) I1010 20:58:41.910727 3550 tablet_service.cc:619] Processing CreateTablet for tablet efe8fb09a1234c9fb1dc99948737a894 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (7), range: [, ) I1010 20:58:41.910800 2963 tablet_service.cc:619] Processing CreateTablet for tablet 8da5904fd56d441ea29e61eb9d2b0b2f (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (11), range: [, ) I1010 20:58:41.910827 3549 tablet_service.cc:619] Processing CreateTablet for tablet ed24ca4817744964991178277119b367 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (17), range: [, ) I1010 20:58:41.910879 2962 tablet_service.cc:619] Processing CreateTablet for tablet 84fcfff9fdef4acf8c2b47f36330567f (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (13), range: [, ) I1010 20:58:41.910886 2612 tablet_service.cc:619] Processing CreateTablet for tablet 17a4ac4e4e164cebb4d88f59511cf926 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (2), range: [, ) I1010 20:58:41.910944 3548 tablet_service.cc:619] Processing CreateTablet for tablet 7836528275cb45aab827361611bb3bbe (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (18), range: [, ) I1010 20:58:41.911054 3547 tablet_service.cc:619] Processing CreateTablet for tablet 970154c84d6849cc83af1d9e5ad6980d (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (19), range: [, ) I1010 20:58:41.911165 3430 ts_tablet_manager.cc:766] Registered tablet f689db1eddb54b1f85b7df38d14534b4 I1010 20:58:41.911275 2962 ts_tablet_manager.cc:766] Registered tablet 84fcfff9fdef4acf8c2b47f36330567f I1010 20:58:41.911283 2490 ts_tablet_manager.cc:766] Registered tablet 19f492d94e864f1ebd1c70810b0470f7 I1010 20:58:41.911316 2961 tablet_service.cc:619] Processing CreateTablet for tablet de9d164718e74d3db902ea603ab32407 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (16), range: [, ) I1010 20:58:41.911324 3198 ts_tablet_manager.cc:766] Registered tablet 90fa7ae3643e4575a6be1c4936b6cc11 I1010 20:58:41.911375 3196 ts_tablet_manager.cc:766] Registered tablet f689db1eddb54b1f85b7df38d14534b4 I1010 20:58:41.911731 2960 tablet_service.cc:619] Processing CreateTablet for tablet 970154c84d6849cc83af1d9e5ad6980d (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (19), range: [, ) I1010 20:58:41.911821 2612 ts_tablet_manager.cc:766] Registered tablet 17a4ac4e4e164cebb4d88f59511cf926 I1010 20:58:41.911885 2611 tablet_service.cc:619] Processing CreateTablet for tablet 05a5666441ec49d792222a3e8674edf7 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (3), range: [, ) I1010 20:58:41.912159 2610 tablet_service.cc:619] Processing CreateTablet for tablet 95e7c143b0524a9b9ade2cfcc081e679 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (5), range: [, ) I1010 20:58:41.912192 2966 ts_tablet_manager.cc:766] Registered tablet 949b26dfeadc4de38868bf4ed994d184 I1010 20:58:41.912297 3315 ts_tablet_manager.cc:766] Registered tablet f2289fb138ea4a5ab6c02803c4e9aa51 I1010 20:58:41.912461 3314 ts_tablet_manager.cc:766] Registered tablet 4c179cfa32ee4128ad57d34140e5320f I1010 20:58:41.912551 2961 ts_tablet_manager.cc:766] Registered tablet de9d164718e74d3db902ea603ab32407 I1010 20:58:41.912564 2609 tablet_service.cc:619] Processing CreateTablet for tablet 4aa163baddb74c4998157964dd55b1f4 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (8), range: [, ) I1010 20:58:41.912588 3551 ts_tablet_manager.cc:766] Registered tablet 17a4ac4e4e164cebb4d88f59511cf926 I1010 20:58:41.912667 2610 ts_tablet_manager.cc:766] Registered tablet 95e7c143b0524a9b9ade2cfcc081e679 I1010 20:58:41.912710 2963 ts_tablet_manager.cc:766] Registered tablet 8da5904fd56d441ea29e61eb9d2b0b2f I1010 20:58:41.912716 2611 ts_tablet_manager.cc:766] Registered tablet 05a5666441ec49d792222a3e8674edf7 I1010 20:58:41.908664 2724 ts_tablet_manager.cc:766] Registered tablet 4c179cfa32ee4128ad57d34140e5320f I1010 20:58:41.912763 2964 ts_tablet_manager.cc:766] Registered tablet efe8fb09a1234c9fb1dc99948737a894 I1010 20:58:41.912784 2608 tablet_service.cc:619] Processing CreateTablet for tablet 981834f5e88f4585b6420fca7ddcb289 (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (10), range: [, ) I1010 20:58:41.912806 2960 ts_tablet_manager.cc:766] Registered tablet 970154c84d6849cc83af1d9e5ad6980d I1010 20:58:41.912999 2609 ts_tablet_manager.cc:766] Registered tablet 4aa163baddb74c4998157964dd55b1f4 I1010 20:58:41.913004 3648 tablet_bootstrap.cc:542] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.913141 3547 ts_tablet_manager.cc:766] Registered tablet 970154c84d6849cc83af1d9e5ad6980d I1010 20:58:41.913146 3648 tablet_bootstrap.cc:482] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377: No blocks or log segments found. Creating new log. I1010 20:58:41.913153 2608 ts_tablet_manager.cc:766] Registered tablet 981834f5e88f4585b6420fca7ddcb289 I1010 20:58:41.913178 3548 ts_tablet_manager.cc:766] Registered tablet 7836528275cb45aab827361611bb3bbe I1010 20:58:41.913197 2607 tablet_service.cc:619] Processing CreateTablet for tablet 84fcfff9fdef4acf8c2b47f36330567f (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (13), range: [, ) I1010 20:58:41.913297 3648 log.cc:347] Log is configured to *not* fsync() on all Append() calls I1010 20:58:41.913487 3550 ts_tablet_manager.cc:766] Registered tablet efe8fb09a1234c9fb1dc99948737a894 I1010 20:58:41.913580 2607 ts_tablet_manager.cc:766] Registered tablet 84fcfff9fdef4acf8c2b47f36330567f I1010 20:58:41.913779 3652 ts_tablet_manager.cc:632] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4: Bootstrapping tablet I1010 20:58:41.913842 3549 ts_tablet_manager.cc:766] Registered tablet ed24ca4817744964991178277119b367 I1010 20:58:41.914106 3652 tablet_bootstrap.cc:380] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4: Bootstrap starting. I1010 20:58:41.914319 3652 tablet_bootstrap.cc:542] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.914460 3652 tablet_bootstrap.cc:482] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4: No blocks or log segments found. Creating new log. I1010 20:58:41.914598 3649 ts_tablet_manager.cc:632] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531: Bootstrapping tablet I1010 20:58:41.914790 3649 tablet_bootstrap.cc:380] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531: Bootstrap starting. I1010 20:58:41.914906 3649 tablet_bootstrap.cc:542] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.915189 3651 ts_tablet_manager.cc:632] T 4aa163baddb74c4998157964dd55b1f4 P 84bde34331b94c4ea003e48f9317f792: Bootstrapping tablet I1010 20:58:41.915238 3647 ts_tablet_manager.cc:632] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93: Bootstrapping tablet I1010 20:58:41.915400 3647 tablet_bootstrap.cc:380] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93: Bootstrap starting. I1010 20:58:41.915567 3649 tablet_bootstrap.cc:482] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531: No blocks or log segments found. Creating new log. I1010 20:58:41.915603 3651 tablet_bootstrap.cc:380] T 4aa163baddb74c4998157964dd55b1f4 P 84bde34331b94c4ea003e48f9317f792: Bootstrap starting. I1010 20:58:41.915616 3652 log.cc:347] Log is configured to *not* fsync() on all Append() calls I1010 20:58:41.915706 3649 log.cc:347] Log is configured to *not* fsync() on all Append() calls I1010 20:58:41.915766 3656 ts_tablet_manager.cc:632] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6: Bootstrapping tablet I1010 20:58:41.915873 3655 ts_tablet_manager.cc:632] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61: Bootstrapping tablet I1010 20:58:41.915936 3656 tablet_bootstrap.cc:380] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6: Bootstrap starting. I1010 20:58:41.916015 3655 tablet_bootstrap.cc:380] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61: Bootstrap starting. I1010 20:58:41.916075 3647 tablet_bootstrap.cc:542] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.916131 3655 tablet_bootstrap.cc:542] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.916209 3651 tablet_bootstrap.cc:542] T 4aa163baddb74c4998157964dd55b1f4 P 84bde34331b94c4ea003e48f9317f792: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.916239 3655 tablet_bootstrap.cc:482] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61: No blocks or log segments found. Creating new log. I1010 20:58:41.916333 3647 tablet_bootstrap.cc:482] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93: No blocks or log segments found. Creating new log. I1010 20:58:41.916368 3655 log.cc:347] Log is configured to *not* fsync() on all Append() calls I1010 20:58:41.916452 3647 log.cc:347] Log is configured to *not* fsync() on all Append() calls I1010 20:58:41.916596 3651 tablet_bootstrap.cc:482] T 4aa163baddb74c4998157964dd55b1f4 P 84bde34331b94c4ea003e48f9317f792: No blocks or log segments found. Creating new log. I1010 20:58:41.916726 3656 tablet_bootstrap.cc:542] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.916851 3656 tablet_bootstrap.cc:482] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6: No blocks or log segments found. Creating new log. I1010 20:58:41.917022 3656 log.cc:347] Log is configured to *not* fsync() on all Append() calls I1010 20:58:41.917028 3651 log.cc:347] Log is configured to *not* fsync() on all Append() calls I1010 20:58:41.917093 3655 tablet_bootstrap.cc:380] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61: No bootstrap required, opened a new log I1010 20:58:41.917115 3655 ts_tablet_manager.cc:637] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:41.917501 3647 tablet_bootstrap.cc:380] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93: No bootstrap required, opened a new log I1010 20:58:41.917524 3647 ts_tablet_manager.cc:637] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s I1010 20:58:41.917613 3655 raft_consensus.cc:275] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } I1010 20:58:41.917636 3655 raft_consensus.cc:306] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:41.917649 3655 raft_consensus.cc:502] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1031512f5120422e9e22a0afabd78f61, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:41.917660 3655 consensus_queue.cc:168] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:41.917714 3651 tablet_bootstrap.cc:380] T 4aa163baddb74c4998157964dd55b1f4 P 84bde34331b94c4ea003e48f9317f792: No bootstrap required, opened a new log I1010 20:58:41.917733 3651 ts_tablet_manager.cc:637] T 4aa163baddb74c4998157964dd55b1f4 P 84bde34331b94c4ea003e48f9317f792: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s I1010 20:58:41.917771 3656 tablet_bootstrap.cc:380] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6: No bootstrap required, opened a new log I1010 20:58:41.917798 3656 ts_tablet_manager.cc:637] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s I1010 20:58:41.917994 2675 heartbeater.cc:365] Master 127.0.0.1:51244 was elected leader, sending a full tablet report... I1010 20:58:41.918069 3647 raft_consensus.cc:275] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } I1010 20:58:41.918097 3647 raft_consensus.cc:306] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:41.918121 3647 raft_consensus.cc:502] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de6cfa864d64c58bf4eafb689517c93, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:41.918134 3647 consensus_queue.cc:168] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:41.918134 3655 ts_tablet_manager.cc:664] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:41.918149 3655 ts_tablet_manager.cc:632] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61: Bootstrapping tablet I1010 20:58:41.918210 3656 raft_consensus.cc:275] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } I1010 20:58:41.918228 3656 raft_consensus.cc:306] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:41.918239 3656 raft_consensus.cc:502] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 580cbaf7092444dfbc6fb344da88d7b6, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:41.918253 3656 consensus_queue.cc:168] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:41.918282 3655 tablet_bootstrap.cc:380] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61: Bootstrap starting. I1010 20:58:41.918326 2909 heartbeater.cc:365] Master 127.0.0.1:51244 was elected leader, sending a full tablet report... I1010 20:58:41.918397 3655 tablet_bootstrap.cc:542] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.918467 3647 ts_tablet_manager.cc:664] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:41.918489 3647 ts_tablet_manager.cc:632] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93: Bootstrapping tablet I1010 20:58:41.918499 3655 tablet_bootstrap.cc:482] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61: No blocks or log segments found. Creating new log. I1010 20:58:41.918608 3647 tablet_bootstrap.cc:380] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93: Bootstrap starting. I1010 20:58:41.919589 3647 tablet_bootstrap.cc:542] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.919632 3649 tablet_bootstrap.cc:380] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531: No bootstrap required, opened a new log I1010 20:58:41.919651 3649 ts_tablet_manager.cc:637] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.000s I1010 20:58:41.919690 3647 tablet_bootstrap.cc:482] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93: No blocks or log segments found. Creating new log. I1010 20:58:41.919744 3614 heartbeater.cc:365] Master 127.0.0.1:51244 was elected leader, sending a full tablet report... I1010 20:58:41.919751 3656 ts_tablet_manager.cc:664] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s I1010 20:58:41.919831 3656 ts_tablet_manager.cc:632] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6: Bootstrapping tablet I1010 20:58:41.919930 3651 raft_consensus.cc:275] T 4aa163baddb74c4998157964dd55b1f4 P 84bde34331b94c4ea003e48f9317f792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } I1010 20:58:41.919981 3656 tablet_bootstrap.cc:380] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6: Bootstrap starting. I1010 20:58:41.919981 3651 raft_consensus.cc:306] T 4aa163baddb74c4998157964dd55b1f4 P 84bde34331b94c4ea003e48f9317f792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:41.919997 3651 raft_consensus.cc:502] T 4aa163baddb74c4998157964dd55b1f4 P 84bde34331b94c4ea003e48f9317f792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84bde34331b94c4ea003e48f9317f792, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:41.920017 3651 consensus_queue.cc:168] T 4aa163baddb74c4998157964dd55b1f4 P 84bde34331b94c4ea003e48f9317f792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:41.920090 3649 raft_consensus.cc:275] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } I1010 20:58:41.920097 3656 tablet_bootstrap.cc:542] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.920109 3649 raft_consensus.cc:306] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:41.920120 3649 raft_consensus.cc:502] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ef4317e50042ba95e517992caa6531, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:41.920128 3649 consensus_queue.cc:168] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:41.920191 3656 tablet_bootstrap.cc:482] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6: No blocks or log segments found. Creating new log. I1010 20:58:41.920192 2557 heartbeater.cc:365] Master 127.0.0.1:51244 was elected leader, sending a full tablet report... I1010 20:58:41.920322 3651 ts_tablet_manager.cc:664] T 4aa163baddb74c4998157964dd55b1f4 P 84bde34331b94c4ea003e48f9317f792: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s I1010 20:58:41.920338 3651 ts_tablet_manager.cc:632] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792: Bootstrapping tablet I1010 20:58:41.920485 3651 tablet_bootstrap.cc:380] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792: Bootstrap starting. I1010 20:58:41.920650 3651 tablet_bootstrap.cc:542] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.920791 3651 tablet_bootstrap.cc:482] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792: No blocks or log segments found. Creating new log. I1010 20:58:41.920817 3656 tablet_bootstrap.cc:380] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6: No bootstrap required, opened a new log I1010 20:58:41.920851 3656 ts_tablet_manager.cc:637] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:41.920966 3146 heartbeater.cc:365] Master 127.0.0.1:51244 was elected leader, sending a full tablet report... I1010 20:58:41.921062 3649 ts_tablet_manager.cc:664] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:41.921077 3649 ts_tablet_manager.cc:632] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531: Bootstrapping tablet I1010 20:58:41.921159 3656 raft_consensus.cc:275] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } I1010 20:58:41.921176 3656 raft_consensus.cc:306] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:41.921185 3656 raft_consensus.cc:502] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 580cbaf7092444dfbc6fb344da88d7b6, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:41.921195 3649 tablet_bootstrap.cc:380] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531: Bootstrap starting. I1010 20:58:41.921198 3656 consensus_queue.cc:168] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:41.921310 3649 tablet_bootstrap.cc:542] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.921340 3656 ts_tablet_manager.cc:664] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.921360 3656 ts_tablet_manager.cc:632] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6: Bootstrapping tablet I1010 20:58:41.921407 3649 tablet_bootstrap.cc:482] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531: No blocks or log segments found. Creating new log. I1010 20:58:41.921469 3656 tablet_bootstrap.cc:380] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6: Bootstrap starting. I1010 20:58:41.921578 3656 tablet_bootstrap.cc:542] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.921691 3656 tablet_bootstrap.cc:482] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6: No blocks or log segments found. Creating new log. I1010 20:58:41.921957 3651 tablet_bootstrap.cc:380] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792: No bootstrap required, opened a new log I1010 20:58:41.921980 3651 ts_tablet_manager.cc:637] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s I1010 20:58:41.922235 3656 tablet_bootstrap.cc:380] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6: No bootstrap required, opened a new log I1010 20:58:41.922279 3656 ts_tablet_manager.cc:637] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:41.922559 3654 ts_tablet_manager.cc:632] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c: Bootstrapping tablet I1010 20:58:41.922585 3651 raft_consensus.cc:275] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } I1010 20:58:41.922644 3651 raft_consensus.cc:306] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:41.922693 3651 raft_consensus.cc:502] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84bde34331b94c4ea003e48f9317f792, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:41.922729 3656 raft_consensus.cc:275] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:58:41.922732 3651 consensus_queue.cc:168] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:41.922793 3648 tablet_bootstrap.cc:380] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377: No bootstrap required, opened a new log I1010 20:58:41.922794 3647 tablet_bootstrap.cc:380] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93: No bootstrap required, opened a new log I1010 20:58:41.922803 3653 ts_tablet_manager.cc:632] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44: Bootstrapping tablet I1010 20:58:41.922812 3648 ts_tablet_manager.cc:637] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.000s I1010 20:58:41.922813 3647 ts_tablet_manager.cc:637] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.000s I1010 20:58:41.923002 3653 tablet_bootstrap.cc:380] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44: Bootstrap starting. I1010 20:58:41.923032 3651 ts_tablet_manager.cc:664] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:41.923046 3651 ts_tablet_manager.cc:632] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792: Bootstrapping tablet I1010 20:58:41.923130 3648 raft_consensus.cc:275] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } I1010 20:58:41.923143 3648 raft_consensus.cc:306] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:41.923153 3648 raft_consensus.cc:502] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d15ccd64ec49be8396ed78e2a88377, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:41.923152 3651 tablet_bootstrap.cc:380] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792: Bootstrap starting. I1010 20:58:41.923161 3648 consensus_queue.cc:168] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:41.923254 3648 ts_tablet_manager.cc:664] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.923269 3648 ts_tablet_manager.cc:632] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377: Bootstrapping tablet I1010 20:58:41.923387 3648 tablet_bootstrap.cc:380] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377: Bootstrap starting. I1010 20:58:41.924383 3651 tablet_bootstrap.cc:542] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.924391 3653 tablet_bootstrap.cc:542] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.924394 3648 tablet_bootstrap.cc:542] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.924482 3651 tablet_bootstrap.cc:482] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792: No blocks or log segments found. Creating new log. I1010 20:58:41.924487 3653 tablet_bootstrap.cc:482] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44: No blocks or log segments found. Creating new log. I1010 20:58:41.924504 3648 tablet_bootstrap.cc:482] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377: No blocks or log segments found. Creating new log. I1010 20:58:41.924732 2792 heartbeater.cc:365] Master 127.0.0.1:51244 was elected leader, sending a full tablet report... I1010 20:58:41.924734 3653 log.cc:347] Log is configured to *not* fsync() on all Append() calls I1010 20:58:41.924762 3647 raft_consensus.cc:275] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:58:41.924774 3647 raft_consensus.cc:306] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:41.924783 3647 raft_consensus.cc:502] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de6cfa864d64c58bf4eafb689517c93, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:41.924793 3647 consensus_queue.cc:168] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:41.924940 3650 ts_tablet_manager.cc:632] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978: Bootstrapping tablet I1010 20:58:41.925071 3654 tablet_bootstrap.cc:380] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c: Bootstrap starting. I1010 20:58:41.925246 3656 raft_consensus.cc:306] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:41.925292 3650 tablet_bootstrap.cc:380] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978: Bootstrap starting. I1010 20:58:41.925302 3656 raft_consensus.cc:502] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 580cbaf7092444dfbc6fb344da88d7b6, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:41.925324 3656 consensus_queue.cc:168] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:41.925451 3650 tablet_bootstrap.cc:542] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.925561 3650 tablet_bootstrap.cc:482] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978: No blocks or log segments found. Creating new log. I1010 20:58:41.925647 3654 tablet_bootstrap.cc:542] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.925683 3650 log.cc:347] Log is configured to *not* fsync() on all Append() calls I1010 20:58:41.925782 3654 tablet_bootstrap.cc:482] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c: No blocks or log segments found. Creating new log. I1010 20:58:41.925789 3647 ts_tablet_manager.cc:664] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s I1010 20:58:41.925806 3647 ts_tablet_manager.cc:632] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93: Bootstrapping tablet I1010 20:58:41.925914 3654 log.cc:347] Log is configured to *not* fsync() on all Append() calls I1010 20:58:41.925927 3647 tablet_bootstrap.cc:380] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93: Bootstrap starting. I1010 20:58:41.926002 3656 ts_tablet_manager.cc:664] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6: Time spent starting tablet: real 0.004s user 0.000s sys 0.000s I1010 20:58:41.926017 3656 ts_tablet_manager.cc:632] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6: Bootstrapping tablet I1010 20:58:41.926028 3647 tablet_bootstrap.cc:542] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.926122 3647 tablet_bootstrap.cc:482] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93: No blocks or log segments found. Creating new log. I1010 20:58:41.926131 3656 tablet_bootstrap.cc:380] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6: Bootstrap starting. I1010 20:58:41.926141 3649 tablet_bootstrap.cc:380] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531: No bootstrap required, opened a new log I1010 20:58:41.926157 3649 ts_tablet_manager.cc:637] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.000s I1010 20:58:41.926232 3656 tablet_bootstrap.cc:542] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.926254 3655 tablet_bootstrap.cc:380] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61: No bootstrap required, opened a new log I1010 20:58:41.926271 3655 ts_tablet_manager.cc:637] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.000s I1010 20:58:41.926333 3656 tablet_bootstrap.cc:482] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6: No blocks or log segments found. Creating new log. I1010 20:58:41.926476 3651 tablet_bootstrap.cc:380] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792: No bootstrap required, opened a new log I1010 20:58:41.926492 3651 ts_tablet_manager.cc:637] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792: Time spent bootstrapping tablet: real 0.003s user 0.004s sys 0.000s I1010 20:58:41.926704 3648 tablet_bootstrap.cc:380] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377: No bootstrap required, opened a new log I1010 20:58:41.926722 3648 ts_tablet_manager.cc:637] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s I1010 20:58:41.926899 3654 tablet_bootstrap.cc:380] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c: No bootstrap required, opened a new log I1010 20:58:41.926925 3654 ts_tablet_manager.cc:637] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.000s I1010 20:58:41.927956 3655 raft_consensus.cc:275] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } I1010 20:58:41.927991 3655 raft_consensus.cc:306] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:41.928004 3655 raft_consensus.cc:502] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1031512f5120422e9e22a0afabd78f61, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:41.928014 3655 consensus_queue.cc:168] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:41.928107 3651 raft_consensus.cc:275] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } I1010 20:58:41.928112 3656 tablet_bootstrap.cc:380] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6: No bootstrap required, opened a new log I1010 20:58:41.928119 3651 raft_consensus.cc:306] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:41.928128 3651 raft_consensus.cc:502] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84bde34331b94c4ea003e48f9317f792, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:41.928129 3656 ts_tablet_manager.cc:637] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s I1010 20:58:41.928136 3651 consensus_queue.cc:168] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:41.928228 3655 ts_tablet_manager.cc:664] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s I1010 20:58:41.928246 3655 ts_tablet_manager.cc:632] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61: Bootstrapping tablet I1010 20:58:41.928352 3655 tablet_bootstrap.cc:380] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61: Bootstrap starting. I1010 20:58:41.928354 3654 raft_consensus.cc:275] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:58:41.928372 3654 raft_consensus.cc:306] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:41.928383 3654 raft_consensus.cc:502] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8062d66fd284035ae3bc2567bc11e7c, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:41.928395 3654 consensus_queue.cc:168] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:41.928458 3655 tablet_bootstrap.cc:542] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.928499 3656 raft_consensus.cc:275] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } I1010 20:58:41.928513 3656 raft_consensus.cc:306] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:41.928520 3656 raft_consensus.cc:502] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 580cbaf7092444dfbc6fb344da88d7b6, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:41.928529 3656 consensus_queue.cc:168] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:41.928544 3655 tablet_bootstrap.cc:482] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61: No blocks or log segments found. Creating new log. I1010 20:58:41.928591 3029 heartbeater.cc:365] Master 127.0.0.1:51244 was elected leader, sending a full tablet report... I1010 20:58:41.928715 3654 ts_tablet_manager.cc:664] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s I1010 20:58:41.928716 3651 ts_tablet_manager.cc:664] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s I1010 20:58:41.928735 3654 ts_tablet_manager.cc:632] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c: Bootstrapping tablet I1010 20:58:41.928735 3651 ts_tablet_manager.cc:632] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792: Bootstrapping tablet I1010 20:58:41.928848 3654 tablet_bootstrap.cc:380] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c: Bootstrap starting. I1010 20:58:41.928851 3651 tablet_bootstrap.cc:380] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792: Bootstrap starting. I1010 20:58:41.928947 3651 tablet_bootstrap.cc:542] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.928973 3654 tablet_bootstrap.cc:542] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.929038 3651 tablet_bootstrap.cc:482] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792: No blocks or log segments found. Creating new log. I1010 20:58:41.929083 3654 tablet_bootstrap.cc:482] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c: No blocks or log segments found. Creating new log. I1010 20:58:41.929252 3656 ts_tablet_manager.cc:664] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6: Time spent starting tablet: real 0.001s user 0.000s sys 0.004s I1010 20:58:41.929298 3656 ts_tablet_manager.cc:632] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6: Bootstrapping tablet I1010 20:58:41.929428 3656 tablet_bootstrap.cc:380] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6: Bootstrap starting. I1010 20:58:41.929513 3648 raft_consensus.cc:275] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } I1010 20:58:41.929528 3648 raft_consensus.cc:306] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:41.929538 3648 raft_consensus.cc:502] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d15ccd64ec49be8396ed78e2a88377, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:41.929546 3648 consensus_queue.cc:168] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:41.929571 3656 tablet_bootstrap.cc:542] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.929646 3648 ts_tablet_manager.cc:664] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s I1010 20:58:41.929662 3648 ts_tablet_manager.cc:632] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377: Bootstrapping tablet I1010 20:58:41.929662 3656 tablet_bootstrap.cc:482] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6: No blocks or log segments found. Creating new log. I1010 20:58:41.929839 3648 tablet_bootstrap.cc:380] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377: Bootstrap starting. I1010 20:58:41.929955 3648 tablet_bootstrap.cc:542] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.930054 3648 tablet_bootstrap.cc:482] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377: No blocks or log segments found. Creating new log. I1010 20:58:41.930357 3656 tablet_bootstrap.cc:380] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6: No bootstrap required, opened a new log I1010 20:58:41.930404 3656 ts_tablet_manager.cc:637] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:41.930716 3648 tablet_bootstrap.cc:380] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377: No bootstrap required, opened a new log I1010 20:58:41.930748 3648 ts_tablet_manager.cc:637] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:41.931049 3648 raft_consensus.cc:275] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } I1010 20:58:41.931064 3648 raft_consensus.cc:306] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:41.931073 3648 raft_consensus.cc:502] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d15ccd64ec49be8396ed78e2a88377, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:41.931082 3648 consensus_queue.cc:168] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:41.931205 3648 ts_tablet_manager.cc:664] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.931221 3648 ts_tablet_manager.cc:632] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377: Bootstrapping tablet I1010 20:58:41.931242 3653 tablet_bootstrap.cc:380] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44: No bootstrap required, opened a new log I1010 20:58:41.931258 3653 ts_tablet_manager.cc:637] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.000s I1010 20:58:41.931298 3650 tablet_bootstrap.cc:380] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978: No bootstrap required, opened a new log I1010 20:58:41.931313 3650 ts_tablet_manager.cc:637] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.000s I1010 20:58:41.931337 3648 tablet_bootstrap.cc:380] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377: Bootstrap starting. I1010 20:58:41.931462 3648 tablet_bootstrap.cc:542] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.931546 3648 tablet_bootstrap.cc:482] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377: No blocks or log segments found. Creating new log. I1010 20:58:41.931865 3650 raft_consensus.cc:275] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:58:41.931895 3650 raft_consensus.cc:306] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:41.931908 3650 raft_consensus.cc:502] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42e94fc3adbb419591836d222d946978, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:41.931921 3650 consensus_queue.cc:168] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:41.932065 3655 tablet_bootstrap.cc:380] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61: No bootstrap required, opened a new log I1010 20:58:41.932075 3653 raft_consensus.cc:275] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:58:41.932088 3655 ts_tablet_manager.cc:637] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.000s I1010 20:58:41.932090 3653 raft_consensus.cc:306] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:41.932114 3653 raft_consensus.cc:502] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f623b00f5ae4b9c8e8940692d5dfd44, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:41.932137 3653 consensus_queue.cc:168] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:41.932312 3380 heartbeater.cc:365] Master 127.0.0.1:51244 was elected leader, sending a full tablet report... I1010 20:58:41.932462 3653 ts_tablet_manager.cc:664] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:41.932472 3649 raft_consensus.cc:275] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } I1010 20:58:41.932476 3653 ts_tablet_manager.cc:632] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44: Bootstrapping tablet I1010 20:58:41.932487 3649 raft_consensus.cc:306] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:41.932508 3649 raft_consensus.cc:502] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ef4317e50042ba95e517992caa6531, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:41.932518 3649 consensus_queue.cc:168] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:41.932641 3653 tablet_bootstrap.cc:380] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44: Bootstrap starting. I1010 20:58:41.932771 3653 tablet_bootstrap.cc:542] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.932863 3653 tablet_bootstrap.cc:482] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44: No blocks or log segments found. Creating new log. I1010 20:58:41.933121 3647 tablet_bootstrap.cc:380] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93: No bootstrap required, opened a new log I1010 20:58:41.933145 3647 ts_tablet_manager.cc:637] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.000s I1010 20:58:41.933329 3651 tablet_bootstrap.cc:380] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792: No bootstrap required, opened a new log I1010 20:58:41.933344 3653 tablet_bootstrap.cc:380] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44: No bootstrap required, opened a new log I1010 20:58:41.933349 3651 ts_tablet_manager.cc:637] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.000s I1010 20:58:41.933363 3653 ts_tablet_manager.cc:637] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:41.933508 3650 ts_tablet_manager.cc:664] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s I1010 20:58:41.933523 3650 ts_tablet_manager.cc:632] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978: Bootstrapping tablet I1010 20:58:41.933632 3651 raft_consensus.cc:275] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } I1010 20:58:41.933645 3650 tablet_bootstrap.cc:380] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978: Bootstrap starting. I1010 20:58:41.933655 3651 raft_consensus.cc:306] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:41.933667 3651 raft_consensus.cc:502] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84bde34331b94c4ea003e48f9317f792, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:41.933676 3651 consensus_queue.cc:168] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:41.933686 3649 ts_tablet_manager.cc:664] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531: Time spent starting tablet: real 0.008s user 0.000s sys 0.000s I1010 20:58:41.933702 3649 ts_tablet_manager.cc:632] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531: Bootstrapping tablet I1010 20:58:41.933792 3650 tablet_bootstrap.cc:542] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.933817 3651 ts_tablet_manager.cc:664] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.933832 3651 ts_tablet_manager.cc:632] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792: Bootstrapping tablet I1010 20:58:41.933850 3649 tablet_bootstrap.cc:380] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531: Bootstrap starting. I1010 20:58:41.933902 3650 tablet_bootstrap.cc:482] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978: No blocks or log segments found. Creating new log. I1010 20:58:41.934010 3651 tablet_bootstrap.cc:380] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792: Bootstrap starting. I1010 20:58:41.934026 3649 tablet_bootstrap.cc:542] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.934114 3497 heartbeater.cc:365] Master 127.0.0.1:51244 was elected leader, sending a full tablet report... I1010 20:58:41.934123 3649 tablet_bootstrap.cc:482] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531: No blocks or log segments found. Creating new log. I1010 20:58:41.934140 3651 tablet_bootstrap.cc:542] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.934222 3651 tablet_bootstrap.cc:482] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792: No blocks or log segments found. Creating new log. I1010 20:58:41.934222 3652 tablet_bootstrap.cc:380] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4: No bootstrap required, opened a new log I1010 20:58:41.934243 3652 ts_tablet_manager.cc:637] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.000s I1010 20:58:41.934293 3647 raft_consensus.cc:275] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } I1010 20:58:41.934308 3647 raft_consensus.cc:306] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:41.934317 3647 raft_consensus.cc:502] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de6cfa864d64c58bf4eafb689517c93, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:41.934327 3647 consensus_queue.cc:168] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:41.934367 3648 tablet_bootstrap.cc:380] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377: No bootstrap required, opened a new log I1010 20:58:41.934384 3648 ts_tablet_manager.cc:637] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s I1010 20:58:41.934398 3655 raft_consensus.cc:275] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } I1010 20:58:41.934412 3655 raft_consensus.cc:306] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:41.934430 3655 raft_consensus.cc:502] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1031512f5120422e9e22a0afabd78f61, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:41.934442 3655 consensus_queue.cc:168] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:41.934486 3654 tablet_bootstrap.cc:380] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c: No bootstrap required, opened a new log I1010 20:58:41.934502 3654 ts_tablet_manager.cc:637] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.000s I1010 20:58:41.934757 3647 ts_tablet_manager.cc:664] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s I1010 20:58:41.934777 3647 ts_tablet_manager.cc:632] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93: Bootstrapping tablet I1010 20:58:41.934808 3648 raft_consensus.cc:275] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } I1010 20:58:41.934814 3652 raft_consensus.cc:275] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } I1010 20:58:41.934821 3648 raft_consensus.cc:306] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:41.934829 3652 raft_consensus.cc:306] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:41.934830 3648 raft_consensus.cc:502] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d15ccd64ec49be8396ed78e2a88377, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:41.934839 3648 consensus_queue.cc:168] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:41.934840 3652 raft_consensus.cc:502] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1380a27d6b824575a3a5e139bee139f4, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:41.934851 3652 consensus_queue.cc:168] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:41.934903 3647 tablet_bootstrap.cc:380] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93: Bootstrap starting. I1010 20:58:41.935039 3647 tablet_bootstrap.cc:542] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.935140 3647 tablet_bootstrap.cc:482] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93: No blocks or log segments found. Creating new log. I1010 20:58:41.936529 3655 ts_tablet_manager.cc:664] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s I1010 20:58:41.936539 3653 raft_consensus.cc:275] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } I1010 20:58:41.936547 3655 ts_tablet_manager.cc:632] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61: Bootstrapping tablet I1010 20:58:41.936553 3653 raft_consensus.cc:306] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:41.936563 3653 raft_consensus.cc:502] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f623b00f5ae4b9c8e8940692d5dfd44, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:41.936573 3653 consensus_queue.cc:168] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:41.936694 3655 tablet_bootstrap.cc:380] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61: Bootstrap starting. I1010 20:58:41.936856 3655 tablet_bootstrap.cc:542] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.936980 3655 tablet_bootstrap.cc:482] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61: No blocks or log segments found. Creating new log. I1010 20:58:41.937206 3654 raft_consensus.cc:275] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } I1010 20:58:41.937222 3654 raft_consensus.cc:306] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:41.937232 3654 raft_consensus.cc:502] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8062d66fd284035ae3bc2567bc11e7c, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:41.937240 3654 consensus_queue.cc:168] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:41.937386 3652 ts_tablet_manager.cc:664] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s I1010 20:58:41.937403 3652 ts_tablet_manager.cc:632] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4: Bootstrapping tablet I1010 20:58:41.937537 3652 tablet_bootstrap.cc:380] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4: Bootstrap starting. I1010 20:58:41.937610 3656 raft_consensus.cc:275] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:58:41.937636 3656 raft_consensus.cc:306] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:41.937646 3656 raft_consensus.cc:502] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 580cbaf7092444dfbc6fb344da88d7b6, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:41.937655 3656 consensus_queue.cc:168] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:41.938665 3654 ts_tablet_manager.cc:664] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c: Time spent starting tablet: real 0.004s user 0.000s sys 0.000s I1010 20:58:41.938690 3652 tablet_bootstrap.cc:542] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.938709 3654 ts_tablet_manager.cc:632] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c: Bootstrapping tablet I1010 20:58:41.938786 3652 tablet_bootstrap.cc:482] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4: No blocks or log segments found. Creating new log. I1010 20:58:41.938829 3654 tablet_bootstrap.cc:380] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c: Bootstrap starting. I1010 20:58:41.938920 3263 heartbeater.cc:365] Master 127.0.0.1:51244 was elected leader, sending a full tablet report... I1010 20:58:41.938935 3654 tablet_bootstrap.cc:542] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.939029 3654 tablet_bootstrap.cc:482] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c: No blocks or log segments found. Creating new log. I1010 20:58:41.939098 3650 tablet_bootstrap.cc:380] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978: No bootstrap required, opened a new log I1010 20:58:41.939117 3650 ts_tablet_manager.cc:637] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.000s I1010 20:58:41.939172 3649 tablet_bootstrap.cc:380] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531: No bootstrap required, opened a new log I1010 20:58:41.939188 3649 ts_tablet_manager.cc:637] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s I1010 20:58:41.939509 3656 ts_tablet_manager.cc:664] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6: Time spent starting tablet: real 0.009s user 0.000s sys 0.000s I1010 20:58:41.939568 3647 tablet_bootstrap.cc:380] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93: No bootstrap required, opened a new log I1010 20:58:41.939589 3647 ts_tablet_manager.cc:637] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.000s I1010 20:58:41.939908 3650 raft_consensus.cc:275] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } I1010 20:58:41.939925 3650 raft_consensus.cc:306] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:41.939936 3649 raft_consensus.cc:275] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } I1010 20:58:41.939944 3650 raft_consensus.cc:502] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42e94fc3adbb419591836d222d946978, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:41.939950 3649 raft_consensus.cc:306] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:41.939954 3650 consensus_queue.cc:168] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:41.939961 3649 raft_consensus.cc:502] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ef4317e50042ba95e517992caa6531, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:41.939971 3649 consensus_queue.cc:168] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:41.940045 3647 raft_consensus.cc:275] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:58:41.940063 3647 raft_consensus.cc:306] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:41.940073 3647 raft_consensus.cc:502] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de6cfa864d64c58bf4eafb689517c93, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:41.940083 3647 consensus_queue.cc:168] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:41.940187 3649 ts_tablet_manager.cc:664] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:41.940217 3649 ts_tablet_manager.cc:632] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531: Bootstrapping tablet I1010 20:58:41.940218 3650 ts_tablet_manager.cc:664] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:41.940232 3650 ts_tablet_manager.cc:632] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978: Bootstrapping tablet I1010 20:58:41.940343 3649 tablet_bootstrap.cc:380] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531: Bootstrap starting. I1010 20:58:41.940359 3650 tablet_bootstrap.cc:380] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978: Bootstrap starting. I1010 20:58:41.940459 3649 tablet_bootstrap.cc:542] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.940460 3650 tablet_bootstrap.cc:542] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.940553 3650 tablet_bootstrap.cc:482] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978: No blocks or log segments found. Creating new log. I1010 20:58:41.940562 3649 tablet_bootstrap.cc:482] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531: No blocks or log segments found. Creating new log. I1010 20:58:41.940767 3647 ts_tablet_manager.cc:664] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:41.940784 3647 ts_tablet_manager.cc:632] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93: Bootstrapping tablet I1010 20:58:41.940898 3647 tablet_bootstrap.cc:380] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93: Bootstrap starting. I1010 20:58:41.941012 3647 tablet_bootstrap.cc:542] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.941107 3647 tablet_bootstrap.cc:482] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93: No blocks or log segments found. Creating new log. I1010 20:58:41.941718 3648 ts_tablet_manager.cc:664] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377: Time spent starting tablet: real 0.007s user 0.000s sys 0.004s I1010 20:58:41.941738 3648 ts_tablet_manager.cc:632] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377: Bootstrapping tablet I1010 20:58:41.941850 3648 tablet_bootstrap.cc:380] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377: Bootstrap starting. I1010 20:58:41.941897 3652 tablet_bootstrap.cc:380] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4: No bootstrap required, opened a new log I1010 20:58:41.941915 3652 ts_tablet_manager.cc:637] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.000s I1010 20:58:41.941972 3648 tablet_bootstrap.cc:542] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.942044 3648 tablet_bootstrap.cc:482] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377: No blocks or log segments found. Creating new log. I1010 20:58:41.942170 3652 raft_consensus.cc:275] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:58:41.942181 3652 raft_consensus.cc:306] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:41.942189 3652 raft_consensus.cc:502] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1380a27d6b824575a3a5e139bee139f4, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:41.942196 3652 consensus_queue.cc:168] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:41.942319 3652 ts_tablet_manager.cc:664] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.942332 3652 ts_tablet_manager.cc:632] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4: Bootstrapping tablet I1010 20:58:41.942435 3652 tablet_bootstrap.cc:380] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4: Bootstrap starting. I1010 20:58:41.942912 3652 tablet_bootstrap.cc:542] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.943006 3652 tablet_bootstrap.cc:482] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4: No blocks or log segments found. Creating new log. I1010 20:58:41.943240 3653 ts_tablet_manager.cc:664] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent starting tablet: real 0.010s user 0.000s sys 0.004s I1010 20:58:41.943259 3653 ts_tablet_manager.cc:632] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44: Bootstrapping tablet I1010 20:58:41.943377 3653 tablet_bootstrap.cc:380] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44: Bootstrap starting. I1010 20:58:41.943464 3653 tablet_bootstrap.cc:542] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.943543 3653 tablet_bootstrap.cc:482] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44: No blocks or log segments found. Creating new log. I1010 20:58:41.943732 3651 tablet_bootstrap.cc:380] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792: No bootstrap required, opened a new log I1010 20:58:41.943754 3651 ts_tablet_manager.cc:637] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s I1010 20:58:41.944200 3651 raft_consensus.cc:275] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } I1010 20:58:41.944229 3651 raft_consensus.cc:306] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:41.944241 3651 raft_consensus.cc:502] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84bde34331b94c4ea003e48f9317f792, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:41.944249 3651 consensus_queue.cc:168] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:41.944631 3761 raft_consensus.cc:375] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:41.944658 3761 raft_consensus.cc:2031] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:41.944986 3761 raft_consensus.cc:1981] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.643s I1010 20:58:41.945044 3761 raft_consensus.cc:391] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } I1010 20:58:41.945438 3651 ts_tablet_manager.cc:664] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s I1010 20:58:41.945469 3651 ts_tablet_manager.cc:632] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792: Bootstrapping tablet I1010 20:58:41.945610 3651 tablet_bootstrap.cc:380] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792: Bootstrap starting. I1010 20:58:41.945756 3651 tablet_bootstrap.cc:542] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:41.945870 3651 tablet_bootstrap.cc:482] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792: No blocks or log segments found. Creating new log. I1010 20:58:42.066865 3742 raft_consensus.cc:375] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.066884 3742 raft_consensus.cc:2031] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.067195 3742 raft_consensus.cc:1981] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.525s I1010 20:58:42.067256 3742 raft_consensus.cc:391] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:58:42.085882 3764 raft_consensus.cc:375] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.085901 3764 raft_consensus.cc:2031] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.086154 3764 raft_consensus.cc:1981] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.526s I1010 20:58:42.086190 3764 raft_consensus.cc:391] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } I1010 20:58:42.154217 3720 raft_consensus.cc:375] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.154229 3720 raft_consensus.cc:2031] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.169792 3737 raft_consensus.cc:375] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.169811 3737 raft_consensus.cc:2031] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.170161 3737 raft_consensus.cc:1981] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.648s I1010 20:58:42.170204 3737 raft_consensus.cc:391] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:58:42.274718 3748 raft_consensus.cc:375] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.274734 3748 raft_consensus.cc:2031] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.274958 3748 raft_consensus.cc:1981] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.576s I1010 20:58:42.274994 3748 raft_consensus.cc:391] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } I1010 20:58:42.285156 3673 raft_consensus.cc:375] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.285168 3673 raft_consensus.cc:2031] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.300998 3744 raft_consensus.cc:375] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.301018 3744 raft_consensus.cc:2031] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.301363 3744 raft_consensus.cc:1981] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.590s I1010 20:58:42.301398 3744 raft_consensus.cc:391] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:58:42.308909 3654 tablet_bootstrap.cc:380] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c: No bootstrap required, opened a new log I1010 20:58:42.308912 3655 tablet_bootstrap.cc:380] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61: No bootstrap required, opened a new log I1010 20:58:42.308941 3654 ts_tablet_manager.cc:637] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c: Time spent bootstrapping tablet: real 0.370s user 0.000s sys 0.000s I1010 20:58:42.308944 3655 ts_tablet_manager.cc:637] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61: Time spent bootstrapping tablet: real 0.372s user 0.000s sys 0.000s I1010 20:58:42.309238 3649 tablet_bootstrap.cc:380] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531: No bootstrap required, opened a new log I1010 20:58:42.309260 3649 ts_tablet_manager.cc:637] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531: Time spent bootstrapping tablet: real 0.369s user 0.000s sys 0.000s I1010 20:58:42.309326 3655 raft_consensus.cc:275] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } I1010 20:58:42.309340 3655 raft_consensus.cc:306] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:42.309350 3655 raft_consensus.cc:502] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1031512f5120422e9e22a0afabd78f61, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:42.309358 3655 consensus_queue.cc:168] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:42.309504 3655 ts_tablet_manager.cc:664] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:42.309521 3655 ts_tablet_manager.cc:632] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61: Bootstrapping tablet I1010 20:58:42.309533 3654 raft_consensus.cc:275] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:58:42.309557 3654 raft_consensus.cc:306] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:42.309566 3654 raft_consensus.cc:502] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8062d66fd284035ae3bc2567bc11e7c, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:42.309576 3654 consensus_queue.cc:168] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:42.309639 3655 tablet_bootstrap.cc:380] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61: Bootstrap starting. I1010 20:58:42.309648 3649 raft_consensus.cc:275] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:58:42.309667 3649 raft_consensus.cc:306] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:42.309677 3649 raft_consensus.cc:502] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ef4317e50042ba95e517992caa6531, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:42.309685 3649 consensus_queue.cc:168] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:42.309756 3654 ts_tablet_manager.cc:664] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s I1010 20:58:42.309770 3654 ts_tablet_manager.cc:632] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c: Bootstrapping tablet I1010 20:58:42.309783 3655 tablet_bootstrap.cc:542] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:42.309983 3655 tablet_bootstrap.cc:482] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61: No blocks or log segments found. Creating new log. I1010 20:58:42.309983 3654 tablet_bootstrap.cc:380] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c: Bootstrap starting. I1010 20:58:42.310143 3654 tablet_bootstrap.cc:542] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:42.310287 3654 tablet_bootstrap.cc:482] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c: No blocks or log segments found. Creating new log. I1010 20:58:42.311578 3649 ts_tablet_manager.cc:664] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s I1010 20:58:42.311594 3649 ts_tablet_manager.cc:632] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531: Bootstrapping tablet I1010 20:58:42.311707 3649 tablet_bootstrap.cc:380] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531: Bootstrap starting. I1010 20:58:42.311818 3649 tablet_bootstrap.cc:542] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:42.311911 3649 tablet_bootstrap.cc:482] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531: No blocks or log segments found. Creating new log. I1010 20:58:42.312340 3651 tablet_bootstrap.cc:380] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792: No bootstrap required, opened a new log I1010 20:58:42.312364 3651 ts_tablet_manager.cc:637] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792: Time spent bootstrapping tablet: real 0.367s user 0.000s sys 0.000s I1010 20:58:42.312507 3650 tablet_bootstrap.cc:380] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978: No bootstrap required, opened a new log I1010 20:58:42.312543 3650 ts_tablet_manager.cc:637] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978: Time spent bootstrapping tablet: real 0.372s user 0.000s sys 0.004s I1010 20:58:42.312683 3647 tablet_bootstrap.cc:380] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93: No bootstrap required, opened a new log I1010 20:58:42.312711 3647 ts_tablet_manager.cc:637] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93: Time spent bootstrapping tablet: real 0.372s user 0.000s sys 0.004s I1010 20:58:42.312906 3653 tablet_bootstrap.cc:380] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44: No bootstrap required, opened a new log I1010 20:58:42.312937 3653 ts_tablet_manager.cc:637] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent bootstrapping tablet: real 0.370s user 0.000s sys 0.000s I1010 20:58:42.313045 3648 tablet_bootstrap.cc:380] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377: No bootstrap required, opened a new log I1010 20:58:42.313082 3648 ts_tablet_manager.cc:637] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377: Time spent bootstrapping tablet: real 0.371s user 0.000s sys 0.000s I1010 20:58:42.313244 3650 raft_consensus.cc:275] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } I1010 20:58:42.313263 3650 raft_consensus.cc:306] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:42.313272 3650 raft_consensus.cc:502] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42e94fc3adbb419591836d222d946978, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:42.313288 3650 consensus_queue.cc:168] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:42.313314 3647 raft_consensus.cc:275] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:58:42.313331 3647 raft_consensus.cc:306] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:42.313345 3647 raft_consensus.cc:502] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de6cfa864d64c58bf4eafb689517c93, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:42.313354 3647 consensus_queue.cc:168] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:42.313359 3653 raft_consensus.cc:275] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:58:42.313375 3653 raft_consensus.cc:306] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:42.313385 3653 raft_consensus.cc:502] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f623b00f5ae4b9c8e8940692d5dfd44, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:42.313393 3653 consensus_queue.cc:168] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:42.313426 3648 raft_consensus.cc:275] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } I1010 20:58:42.313444 3648 raft_consensus.cc:306] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:42.313454 3648 raft_consensus.cc:502] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d15ccd64ec49be8396ed78e2a88377, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:42.313464 3648 consensus_queue.cc:168] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:42.313498 3650 ts_tablet_manager.cc:664] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:42.313513 3650 ts_tablet_manager.cc:632] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978: Bootstrapping tablet I1010 20:58:42.313550 3647 ts_tablet_manager.cc:664] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:42.313566 3647 ts_tablet_manager.cc:632] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93: Bootstrapping tablet I1010 20:58:42.313638 3650 tablet_bootstrap.cc:380] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978: Bootstrap starting. I1010 20:58:42.313680 3647 tablet_bootstrap.cc:380] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93: Bootstrap starting. I1010 20:58:42.313747 3650 tablet_bootstrap.cc:542] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:42.313766 3647 tablet_bootstrap.cc:542] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:42.313840 3653 ts_tablet_manager.cc:664] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:42.313859 3653 ts_tablet_manager.cc:632] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44: Bootstrapping tablet I1010 20:58:42.313864 3647 tablet_bootstrap.cc:482] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93: No blocks or log segments found. Creating new log. I1010 20:58:42.314007 3653 tablet_bootstrap.cc:380] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44: Bootstrap starting. I1010 20:58:42.314034 3651 raft_consensus.cc:275] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:58:42.314049 3651 raft_consensus.cc:306] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:42.314059 3651 raft_consensus.cc:502] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84bde34331b94c4ea003e48f9317f792, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:42.314069 3651 consensus_queue.cc:168] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:42.314162 3653 tablet_bootstrap.cc:542] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:42.314174 3648 ts_tablet_manager.cc:664] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:42.314189 3648 ts_tablet_manager.cc:632] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377: Bootstrapping tablet I1010 20:58:42.314307 3653 tablet_bootstrap.cc:482] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44: No blocks or log segments found. Creating new log. I1010 20:58:42.314349 3648 tablet_bootstrap.cc:380] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377: Bootstrap starting. I1010 20:58:42.314435 3648 tablet_bootstrap.cc:542] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:42.314533 3648 tablet_bootstrap.cc:482] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377: No blocks or log segments found. Creating new log. I1010 20:58:42.314831 3651 ts_tablet_manager.cc:664] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s I1010 20:58:42.315033 3650 tablet_bootstrap.cc:482] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978: No blocks or log segments found. Creating new log. I1010 20:58:42.315044 3653 tablet_bootstrap.cc:380] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44: No bootstrap required, opened a new log I1010 20:58:42.315062 3653 ts_tablet_manager.cc:637] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:42.315577 3653 raft_consensus.cc:275] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } I1010 20:58:42.315594 3653 raft_consensus.cc:306] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:42.315604 3653 raft_consensus.cc:502] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f623b00f5ae4b9c8e8940692d5dfd44, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:42.315613 3653 consensus_queue.cc:168] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:42.315781 3648 tablet_bootstrap.cc:380] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377: No bootstrap required, opened a new log I1010 20:58:42.315803 3648 ts_tablet_manager.cc:637] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s I1010 20:58:42.315940 3647 tablet_bootstrap.cc:380] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93: No bootstrap required, opened a new log I1010 20:58:42.315959 3647 ts_tablet_manager.cc:637] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s I1010 20:58:42.315959 3652 tablet_bootstrap.cc:380] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4: No bootstrap required, opened a new log I1010 20:58:42.315981 3652 ts_tablet_manager.cc:637] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4: Time spent bootstrapping tablet: real 0.374s user 0.004s sys 0.000s I1010 20:58:42.316076 3653 ts_tablet_manager.cc:664] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:42.316093 3653 ts_tablet_manager.cc:632] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44: Bootstrapping tablet I1010 20:58:42.316221 3653 tablet_bootstrap.cc:380] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44: Bootstrap starting. I1010 20:58:42.316332 3652 raft_consensus.cc:275] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } I1010 20:58:42.316341 3653 tablet_bootstrap.cc:542] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:42.316345 3652 raft_consensus.cc:306] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:42.316355 3652 raft_consensus.cc:502] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1380a27d6b824575a3a5e139bee139f4, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:42.316375 3652 consensus_queue.cc:168] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:42.316457 3653 tablet_bootstrap.cc:482] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44: No blocks or log segments found. Creating new log. I1010 20:58:42.316515 3652 ts_tablet_manager.cc:664] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:42.316532 3652 ts_tablet_manager.cc:632] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4: Bootstrapping tablet I1010 20:58:42.316548 3647 raft_consensus.cc:275] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:58:42.316562 3647 raft_consensus.cc:306] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:42.316572 3647 raft_consensus.cc:502] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de6cfa864d64c58bf4eafb689517c93, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:42.316581 3647 consensus_queue.cc:168] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:42.316632 3649 tablet_bootstrap.cc:380] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531: No bootstrap required, opened a new log I1010 20:58:42.316646 3652 tablet_bootstrap.cc:380] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4: Bootstrap starting. I1010 20:58:42.316653 3649 ts_tablet_manager.cc:637] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.000s I1010 20:58:42.317658 3652 tablet_bootstrap.cc:542] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:42.317755 3652 tablet_bootstrap.cc:482] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4: No blocks or log segments found. Creating new log. I1010 20:58:42.317833 3647 ts_tablet_manager.cc:664] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s I1010 20:58:42.317852 3647 ts_tablet_manager.cc:632] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93: Bootstrapping tablet I1010 20:58:42.317958 3647 tablet_bootstrap.cc:380] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93: Bootstrap starting. I1010 20:58:42.318059 3647 tablet_bootstrap.cc:542] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:42.318153 3647 tablet_bootstrap.cc:482] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93: No blocks or log segments found. Creating new log. I1010 20:58:42.319541 3652 tablet_bootstrap.cc:380] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4: No bootstrap required, opened a new log I1010 20:58:42.319547 3649 raft_consensus.cc:275] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:58:42.319561 3649 raft_consensus.cc:306] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:42.319562 3652 ts_tablet_manager.cc:637] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.004s I1010 20:58:42.319568 3649 raft_consensus.cc:502] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ef4317e50042ba95e517992caa6531, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:42.319576 3649 consensus_queue.cc:168] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:42.319860 3649 ts_tablet_manager.cc:664] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s I1010 20:58:42.319880 3649 ts_tablet_manager.cc:632] T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531: Bootstrapping tablet I1010 20:58:42.319895 3652 raft_consensus.cc:275] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } I1010 20:58:42.319941 3652 raft_consensus.cc:306] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:42.319957 3652 raft_consensus.cc:502] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1380a27d6b824575a3a5e139bee139f4, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:42.319965 3652 consensus_queue.cc:168] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:42.320010 3649 tablet_bootstrap.cc:380] T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531: Bootstrap starting. I1010 20:58:42.320125 3652 ts_tablet_manager.cc:664] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:42.320127 3649 tablet_bootstrap.cc:542] T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:42.320168 3652 ts_tablet_manager.cc:632] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4: Bootstrapping tablet I1010 20:58:42.320236 3649 tablet_bootstrap.cc:482] T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531: No blocks or log segments found. Creating new log. I1010 20:58:42.320334 3652 tablet_bootstrap.cc:380] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4: Bootstrap starting. I1010 20:58:42.320459 3652 tablet_bootstrap.cc:542] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:42.320545 3652 tablet_bootstrap.cc:482] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4: No blocks or log segments found. Creating new log. I1010 20:58:42.320546 3655 tablet_bootstrap.cc:380] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61: No bootstrap required, opened a new log I1010 20:58:42.320577 3655 ts_tablet_manager.cc:637] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.000s I1010 20:58:42.320935 3655 raft_consensus.cc:275] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } I1010 20:58:42.320948 3655 raft_consensus.cc:306] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:42.320958 3655 raft_consensus.cc:502] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1031512f5120422e9e22a0afabd78f61, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:42.320979 3655 consensus_queue.cc:168] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:42.321125 3655 ts_tablet_manager.cc:664] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:42.321146 3655 ts_tablet_manager.cc:632] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61: Bootstrapping tablet I1010 20:58:42.321275 3655 tablet_bootstrap.cc:380] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61: Bootstrap starting. I1010 20:58:42.321341 3649 tablet_bootstrap.cc:380] T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531: No bootstrap required, opened a new log I1010 20:58:42.321360 3649 ts_tablet_manager.cc:637] T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531: Time spent bootstrapping tablet: real 0.001s user 0.004s sys 0.000s I1010 20:58:42.321455 3655 tablet_bootstrap.cc:542] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:42.321550 3655 tablet_bootstrap.cc:482] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61: No blocks or log segments found. Creating new log. I1010 20:58:42.321597 3652 tablet_bootstrap.cc:380] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4: No bootstrap required, opened a new log I1010 20:58:42.321625 3652 ts_tablet_manager.cc:637] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:42.321837 3649 raft_consensus.cc:275] T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:58:42.321853 3649 raft_consensus.cc:306] T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:42.321862 3649 raft_consensus.cc:502] T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ef4317e50042ba95e517992caa6531, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:42.321871 3649 consensus_queue.cc:168] T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:42.321969 3652 raft_consensus.cc:275] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } I1010 20:58:42.322005 3652 raft_consensus.cc:306] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:42.322039 3652 raft_consensus.cc:502] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1380a27d6b824575a3a5e139bee139f4, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:42.322055 3647 tablet_bootstrap.cc:380] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93: No bootstrap required, opened a new log I1010 20:58:42.322072 3652 consensus_queue.cc:168] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:42.322077 3647 ts_tablet_manager.cc:637] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.000s I1010 20:58:42.322142 3649 ts_tablet_manager.cc:664] T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:42.322212 3652 ts_tablet_manager.cc:664] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:42.322458 3647 raft_consensus.cc:275] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } I1010 20:58:42.322474 3647 raft_consensus.cc:306] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:42.322484 3647 raft_consensus.cc:502] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de6cfa864d64c58bf4eafb689517c93, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:42.322494 3647 consensus_queue.cc:168] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:42.322716 3647 ts_tablet_manager.cc:664] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:42.323403 3655 tablet_bootstrap.cc:380] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61: No bootstrap required, opened a new log I1010 20:58:42.323424 3655 ts_tablet_manager.cc:637] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.004s I1010 20:58:42.323602 3654 tablet_bootstrap.cc:380] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c: No bootstrap required, opened a new log I1010 20:58:42.323603 3653 tablet_bootstrap.cc:380] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44: No bootstrap required, opened a new log I1010 20:58:42.323622 3653 ts_tablet_manager.cc:637] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s I1010 20:58:42.323631 3654 ts_tablet_manager.cc:637] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.000s I1010 20:58:42.323629 3648 raft_consensus.cc:275] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:58:42.323642 3648 raft_consensus.cc:306] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:42.323652 3648 raft_consensus.cc:502] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d15ccd64ec49be8396ed78e2a88377, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:42.323659 3648 consensus_queue.cc:168] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:42.323824 3650 tablet_bootstrap.cc:380] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978: No bootstrap required, opened a new log I1010 20:58:42.323840 3650 ts_tablet_manager.cc:637] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.000s I1010 20:58:42.323932 3648 ts_tablet_manager.cc:664] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377: Time spent starting tablet: real 0.008s user 0.000s sys 0.000s I1010 20:58:42.324055 3654 raft_consensus.cc:275] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:58:42.324067 3654 raft_consensus.cc:306] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:42.324076 3654 raft_consensus.cc:502] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8062d66fd284035ae3bc2567bc11e7c, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:42.324084 3654 consensus_queue.cc:168] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:42.324153 3650 raft_consensus.cc:275] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:58:42.324167 3650 raft_consensus.cc:306] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:42.324175 3650 raft_consensus.cc:502] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42e94fc3adbb419591836d222d946978, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:42.324183 3650 consensus_queue.cc:168] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:42.324285 3654 ts_tablet_manager.cc:664] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:42.324301 3654 ts_tablet_manager.cc:632] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c: Bootstrapping tablet I1010 20:58:42.324486 3654 tablet_bootstrap.cc:380] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c: Bootstrap starting. I1010 20:58:42.324609 3654 tablet_bootstrap.cc:542] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:42.324720 3654 tablet_bootstrap.cc:482] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c: No blocks or log segments found. Creating new log. I1010 20:58:42.324879 3650 ts_tablet_manager.cc:664] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:42.324897 3650 ts_tablet_manager.cc:632] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978: Bootstrapping tablet I1010 20:58:42.325009 3650 tablet_bootstrap.cc:380] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978: Bootstrap starting. I1010 20:58:42.325114 3650 tablet_bootstrap.cc:542] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:42.325206 3650 tablet_bootstrap.cc:482] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978: No blocks or log segments found. Creating new log. I1010 20:58:42.325672 3653 raft_consensus.cc:275] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:58:42.325690 3653 raft_consensus.cc:306] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:42.325696 3653 raft_consensus.cc:502] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f623b00f5ae4b9c8e8940692d5dfd44, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:42.325695 3655 raft_consensus.cc:275] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } I1010 20:58:42.325703 3653 consensus_queue.cc:168] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:42.325707 3655 raft_consensus.cc:306] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:42.325716 3655 raft_consensus.cc:502] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1031512f5120422e9e22a0afabd78f61, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:42.325723 3655 consensus_queue.cc:168] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:42.325867 3653 ts_tablet_manager.cc:664] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s I1010 20:58:42.325865 3655 ts_tablet_manager.cc:664] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s I1010 20:58:42.325884 3653 ts_tablet_manager.cc:632] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44: Bootstrapping tablet I1010 20:58:42.325994 3653 tablet_bootstrap.cc:380] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44: Bootstrap starting. I1010 20:58:42.326119 3653 tablet_bootstrap.cc:542] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:42.326205 3653 tablet_bootstrap.cc:482] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44: No blocks or log segments found. Creating new log. I1010 20:58:42.326452 3650 tablet_bootstrap.cc:380] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978: No bootstrap required, opened a new log I1010 20:58:42.326473 3650 ts_tablet_manager.cc:637] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s I1010 20:58:42.326540 3654 tablet_bootstrap.cc:380] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c: No bootstrap required, opened a new log I1010 20:58:42.326560 3654 ts_tablet_manager.cc:637] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.004s I1010 20:58:42.326764 3650 raft_consensus.cc:275] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } I1010 20:58:42.326781 3650 raft_consensus.cc:306] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:42.326792 3650 raft_consensus.cc:502] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42e94fc3adbb419591836d222d946978, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:42.326805 3650 consensus_queue.cc:168] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:42.326874 3654 raft_consensus.cc:275] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } I1010 20:58:42.326891 3654 raft_consensus.cc:306] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:42.326900 3654 raft_consensus.cc:502] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8062d66fd284035ae3bc2567bc11e7c, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:42.326908 3654 consensus_queue.cc:168] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:42.326956 3650 ts_tablet_manager.cc:664] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:42.326975 3650 ts_tablet_manager.cc:632] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978: Bootstrapping tablet I1010 20:58:42.327062 3654 ts_tablet_manager.cc:664] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:42.327081 3654 ts_tablet_manager.cc:632] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c: Bootstrapping tablet I1010 20:58:42.327080 3650 tablet_bootstrap.cc:380] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978: Bootstrap starting. I1010 20:58:42.327188 3650 tablet_bootstrap.cc:542] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:42.327188 3654 tablet_bootstrap.cc:380] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c: Bootstrap starting. I1010 20:58:42.327278 3650 tablet_bootstrap.cc:482] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978: No blocks or log segments found. Creating new log. I1010 20:58:42.327296 3654 tablet_bootstrap.cc:542] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:42.327383 3654 tablet_bootstrap.cc:482] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c: No blocks or log segments found. Creating new log. I1010 20:58:42.327778 3650 tablet_bootstrap.cc:380] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978: No bootstrap required, opened a new log I1010 20:58:42.327802 3650 ts_tablet_manager.cc:637] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:42.327800 3653 tablet_bootstrap.cc:380] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44: No bootstrap required, opened a new log I1010 20:58:42.327818 3653 ts_tablet_manager.cc:637] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s I1010 20:58:42.328980 3654 tablet_bootstrap.cc:380] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c: No bootstrap required, opened a new log I1010 20:58:42.329007 3654 ts_tablet_manager.cc:637] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s I1010 20:58:42.329041 3650 raft_consensus.cc:275] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } I1010 20:58:42.329041 3653 raft_consensus.cc:275] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:58:42.329056 3650 raft_consensus.cc:306] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:42.329056 3653 raft_consensus.cc:306] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:42.329076 3653 raft_consensus.cc:502] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f623b00f5ae4b9c8e8940692d5dfd44, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:42.329076 3650 raft_consensus.cc:502] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42e94fc3adbb419591836d222d946978, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:42.329084 3650 consensus_queue.cc:168] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:42.329084 3653 consensus_queue.cc:168] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:42.329217 3653 ts_tablet_manager.cc:664] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1010 20:58:42.329236 3654 raft_consensus.cc:275] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } I1010 20:58:42.329248 3654 raft_consensus.cc:306] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:42.329257 3654 raft_consensus.cc:502] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8062d66fd284035ae3bc2567bc11e7c, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:42.329270 3654 consensus_queue.cc:168] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:42.329396 3650 ts_tablet_manager.cc:664] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s I1010 20:58:42.329406 3654 ts_tablet_manager.cc:664] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:42.329421 3654 ts_tablet_manager.cc:632] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c: Bootstrapping tablet I1010 20:58:42.329532 3654 tablet_bootstrap.cc:380] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c: Bootstrap starting. I1010 20:58:42.329653 3654 tablet_bootstrap.cc:542] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:42.329741 3654 tablet_bootstrap.cc:482] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c: No blocks or log segments found. Creating new log. I1010 20:58:42.329898 3848 raft_consensus.cc:375] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.329913 3848 raft_consensus.cc:2031] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.339025 3891 raft_consensus.cc:375] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.339040 3891 raft_consensus.cc:2031] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.341497 3654 tablet_bootstrap.cc:380] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c: No bootstrap required, opened a new log I1010 20:58:42.341526 3654 ts_tablet_manager.cc:637] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.000s I1010 20:58:42.341573 3755 raft_consensus.cc:375] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.341612 3755 raft_consensus.cc:2031] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.341814 3654 raft_consensus.cc:275] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } I1010 20:58:42.341838 3654 raft_consensus.cc:306] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:58:42.341845 3654 raft_consensus.cc:502] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8062d66fd284035ae3bc2567bc11e7c, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:42.341862 3654 consensus_queue.cc:168] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:58:42.341995 3654 ts_tablet_manager.cc:664] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1010 20:58:42.343827 3717 raft_consensus.cc:375] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.343840 3717 raft_consensus.cc:2031] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.344060 3717 raft_consensus.cc:1981] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.511s I1010 20:58:42.344094 3717 raft_consensus.cc:391] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } I1010 20:58:42.349930 3779 raft_consensus.cc:375] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.349946 3779 raft_consensus.cc:2031] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.362063 3738 raft_consensus.cc:375] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.362077 3738 raft_consensus.cc:2031] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.362277 3738 raft_consensus.cc:1981] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.564s I1010 20:58:42.362308 3738 raft_consensus.cc:391] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } I1010 20:58:42.402698 3763 raft_consensus.cc:375] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.402712 3763 raft_consensus.cc:2031] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.407985 3780 raft_consensus.cc:375] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.408000 3780 raft_consensus.cc:2031] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.408200 3780 raft_consensus.cc:1981] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.589s I1010 20:58:42.408236 3780 raft_consensus.cc:391] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } I1010 20:58:42.430708 3754 raft_consensus.cc:375] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.430726 3754 raft_consensus.cc:2031] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.439013 3781 raft_consensus.cc:375] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.439028 3781 raft_consensus.cc:2031] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.439317 3781 raft_consensus.cc:1981] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.620s I1010 20:58:42.439348 3781 raft_consensus.cc:391] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:58:42.441063 3678 raft_consensus.cc:375] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.441077 3678 raft_consensus.cc:2031] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.441795 3881 raft_consensus.cc:375] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.441823 3881 raft_consensus.cc:2031] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.452683 3715 raft_consensus.cc:375] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.452698 3715 raft_consensus.cc:2031] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.455122 3794 raft_consensus.cc:375] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.455135 3794 raft_consensus.cc:2031] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.456456 3722 raft_consensus.cc:375] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.456468 3722 raft_consensus.cc:2031] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.504125 3697 raft_consensus.cc:375] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.504140 3697 raft_consensus.cc:2031] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.504778 3694 raft_consensus.cc:375] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.504791 3694 raft_consensus.cc:2031] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.517493 3667 raft_consensus.cc:375] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.517506 3667 raft_consensus.cc:2031] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.522805 3867 raft_consensus.cc:375] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.522822 3867 raft_consensus.cc:2031] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.525313 3692 raft_consensus.cc:375] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.525338 3692 raft_consensus.cc:2031] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.540215 3684 raft_consensus.cc:375] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.540226 3684 raft_consensus.cc:2031] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.544893 3671 raft_consensus.cc:375] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.544905 3671 raft_consensus.cc:2031] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.546808 3753 raft_consensus.cc:375] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.546823 3753 raft_consensus.cc:2031] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.549527 3829 raft_consensus.cc:375] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.549540 3829 raft_consensus.cc:2031] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.615828 3815 raft_consensus.cc:375] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.615841 3815 raft_consensus.cc:2031] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.623962 3853 raft_consensus.cc:375] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.623975 3853 raft_consensus.cc:2031] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.630223 3789 raft_consensus.cc:375] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.630236 3789 raft_consensus.cc:2031] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.637351 3837 raft_consensus.cc:375] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.637374 3837 raft_consensus.cc:2031] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.692097 3672 raft_consensus.cc:375] T 4aa163baddb74c4998157964dd55b1f4 P 84bde34331b94c4ea003e48f9317f792 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.692111 3672 raft_consensus.cc:2031] T 4aa163baddb74c4998157964dd55b1f4 P 84bde34331b94c4ea003e48f9317f792 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.713407 3804 raft_consensus.cc:375] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.713421 3804 raft_consensus.cc:2031] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.744174 3817 raft_consensus.cc:375] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.744189 3817 raft_consensus.cc:2031] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.756724 3870 raft_consensus.cc:375] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.756738 3870 raft_consensus.cc:2031] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.777951 3859 raft_consensus.cc:375] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.777966 3859 raft_consensus.cc:2031] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.800173 3698 raft_consensus.cc:375] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.800186 3698 raft_consensus.cc:2031] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.853337 3713 raft_consensus.cc:375] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.853359 3713 raft_consensus.cc:2031] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.860491 3863 raft_consensus.cc:375] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.860507 3863 raft_consensus.cc:2031] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.866744 3857 raft_consensus.cc:375] T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.866763 3857 raft_consensus.cc:2031] T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.909606 3729 raft_consensus.cc:375] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.909624 3729 raft_consensus.cc:2031] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.953398 3819 raft_consensus.cc:375] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.953411 3819 raft_consensus.cc:2031] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:42.986662 3834 raft_consensus.cc:375] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:42.986690 3834 raft_consensus.cc:2031] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:43.009929 3869 raft_consensus.cc:375] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:43.009948 3869 raft_consensus.cc:2031] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:43.087895 3882 raft_consensus.cc:375] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:43.087913 3882 raft_consensus.cc:2031] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:43.088259 3836 raft_consensus.cc:375] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:43.088287 3836 raft_consensus.cc:2031] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:43.115123 3816 raft_consensus.cc:375] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:43.115147 3816 raft_consensus.cc:2031] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:43.130084 3889 raft_consensus.cc:375] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:43.130100 3889 raft_consensus.cc:2031] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:43.156390 3805 raft_consensus.cc:375] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:43.156404 3805 raft_consensus.cc:2031] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:43.157338 3818 raft_consensus.cc:375] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:43.157362 3818 raft_consensus.cc:2031] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:43.177309 3894 raft_consensus.cc:375] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:43.177321 3894 raft_consensus.cc:2031] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:43.241829 3898 raft_consensus.cc:375] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:43.241842 3898 raft_consensus.cc:2031] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:43.312111 3842 raft_consensus.cc:375] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:43.312129 3842 raft_consensus.cc:2031] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:43.351649 3808 raft_consensus.cc:375] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:43.351665 3808 raft_consensus.cc:2031] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:58:43.368152 3871 raft_consensus.cc:375] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:43.368168 3871 raft_consensus.cc:2031] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [term 0 FOLLOWER]: Advancing to term 1 W1010 20:58:43.664747 3761 consensus_meta.cc:188] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 1.720s user 0.000s sys 0.000s W1010 20:58:43.664747 3742 consensus_meta.cc:188] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 1.597s user 0.000s sys 0.000s W1010 20:58:43.664783 3764 consensus_meta.cc:188] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.579s user 0.000s sys 0.000s W1010 20:58:43.664783 3744 consensus_meta.cc:188] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 1.363s user 0.000s sys 0.000s W1010 20:58:43.664880 3737 consensus_meta.cc:188] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6: Time spent flushing consensus metadata: real 1.495s user 0.000s sys 0.000s W1010 20:58:43.664898 3717 consensus_meta.cc:188] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 1.321s user 0.000s sys 0.000s W1010 20:58:43.664980 3748 consensus_meta.cc:188] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 1.390s user 0.000s sys 0.000s W1010 20:58:43.665180 3891 consensus_meta.cc:188] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 1.326s user 0.000s sys 0.000s I1010 20:58:43.665202 3891 raft_consensus.cc:1981] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.512s W1010 20:58:43.665213 3754 consensus_meta.cc:188] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 1.234s user 0.000s sys 0.000s I1010 20:58:43.665232 3754 raft_consensus.cc:1981] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.565s I1010 20:58:43.665236 3891 raft_consensus.cc:391] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } W1010 20:58:43.665233 3673 consensus_meta.cc:188] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6: Time spent flushing consensus metadata: real 1.380s user 0.000s sys 0.000s I1010 20:58:43.665251 3673 raft_consensus.cc:1981] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.593s I1010 20:58:43.665259 3754 raft_consensus.cc:391] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } I1010 20:58:43.665283 3673 raft_consensus.cc:391] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } W1010 20:58:43.665288 3869 consensus_meta.cc:188] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 0.655s user 0.000s sys 0.000s I1010 20:58:43.665307 3869 raft_consensus.cc:1981] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.571s I1010 20:58:43.665334 3869 raft_consensus.cc:391] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } W1010 20:58:43.665565 3720 consensus_meta.cc:188] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.511s user 0.000s sys 0.000s I1010 20:58:43.665596 3720 raft_consensus.cc:1981] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.558s W1010 20:58:43.665606 3781 consensus_meta.cc:188] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 1.226s user 0.000s sys 0.000s I1010 20:58:43.665627 3720 raft_consensus.cc:391] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } W1010 20:58:43.665735 3738 consensus_meta.cc:188] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 1.303s user 0.000s sys 0.000s W1010 20:58:43.665827 3715 consensus_meta.cc:188] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 1.213s user 0.000s sys 0.000s I1010 20:58:43.665848 3715 raft_consensus.cc:1981] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.559s W1010 20:58:43.665853 3829 consensus_meta.cc:188] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 1.116s user 0.000s sys 0.000s I1010 20:58:43.665885 3829 raft_consensus.cc:1981] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.537s I1010 20:58:43.665885 3715 raft_consensus.cc:391] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } I1010 20:58:43.665913 3829 raft_consensus.cc:391] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } W1010 20:58:43.666013 3697 consensus_meta.cc:188] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 1.162s user 0.000s sys 0.000s I1010 20:58:43.666048 3697 raft_consensus.cc:1981] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.517s I1010 20:58:43.666142 3697 raft_consensus.cc:391] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } W1010 20:58:43.666172 3763 consensus_meta.cc:188] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 1.263s user 0.000s sys 0.000s I1010 20:58:43.666206 3763 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.538s I1010 20:58:43.666249 3763 raft_consensus.cc:391] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } W1010 20:58:43.666532 3848 consensus_meta.cc:188] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 1.337s user 0.000s sys 0.000s I1010 20:58:43.666563 3848 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.618s I1010 20:58:43.666584 3848 raft_consensus.cc:391] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } W1010 20:58:43.666661 3789 consensus_meta.cc:188] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 1.036s user 0.000s sys 0.000s I1010 20:58:43.666710 3789 raft_consensus.cc:1981] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.508s I1010 20:58:43.666751 3789 raft_consensus.cc:391] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } W1010 20:58:43.666942 3780 consensus_meta.cc:188] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 1.259s user 0.000s sys 0.000s W1010 20:58:43.667007 3882 consensus_meta.cc:188] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 0.579s user 0.000s sys 0.000s I1010 20:58:43.667027 3882 raft_consensus.cc:1981] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.538s I1010 20:58:43.667050 3882 raft_consensus.cc:391] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } W1010 20:58:43.667086 3805 consensus_meta.cc:188] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 0.511s user 0.000s sys 0.000s I1010 20:58:43.667105 3805 raft_consensus.cc:1981] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.608s I1010 20:58:43.667131 3805 raft_consensus.cc:391] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:58:43.667176 3894 raft_consensus.cc:1981] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.638s I1010 20:58:43.667201 3894 raft_consensus.cc:391] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } W1010 20:58:43.667210 3678 consensus_meta.cc:188] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 1.226s user 0.000s sys 0.000s I1010 20:58:43.667229 3678 raft_consensus.cc:1981] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.500s I1010 20:58:43.667256 3678 raft_consensus.cc:391] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } I1010 20:58:43.667798 3761 leader_election.cc:223] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 1 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 I1010 20:58:43.667821 3761 leader_election.cc:223] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 1 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:58:43.667820 3742 leader_election.cc:223] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 1 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:58:43.667820 3780 leader_election.cc:223] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 1 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:58:43.667826 3781 leader_election.cc:223] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:58:43.667841 3742 leader_election.cc:223] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 1 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 I1010 20:58:43.667841 3780 leader_election.cc:223] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 1 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 I1010 20:58:43.667845 3781 leader_election.cc:223] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 I1010 20:58:43.668098 3737 leader_election.cc:223] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 1 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:58:43.668118 3737 leader_election.cc:223] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 1 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 I1010 20:58:43.668107 3744 leader_election.cc:223] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:58:43.668138 3744 leader_election.cc:223] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:58:43.668138 3764 leader_election.cc:223] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 I1010 20:58:43.668154 3764 leader_election.cc:223] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:58:43.668287 3738 leader_election.cc:223] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 1 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 I1010 20:58:43.668309 3738 leader_election.cc:223] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 1 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:58:43.668406 3748 leader_election.cc:223] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 1 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:58:43.668453 3748 leader_election.cc:223] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 1 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:58:43.670717 3717 leader_election.cc:223] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 1 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:58:43.670743 3717 leader_election.cc:223] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 1 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:58:43.674268 2512 raft_consensus.cc:1609] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 1: Already voted for candidate 84bde34331b94c4ea003e48f9317f792 in this term. I1010 20:58:43.674365 3039 leader_election.cc:361] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 1 election: Vote denied by peer 84bde34331b94c4ea003e48f9317f792. Message: Invalid argument: T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 1: Already voted for candidate 84bde34331b94c4ea003e48f9317f792 in this term. I1010 20:58:43.674582 3099 raft_consensus.cc:1609] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 in current term 1: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. I1010 20:58:43.674662 2447 leader_election.cc:361] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 1 election: Vote denied by peer 33ef4317e50042ba95e517992caa6531. Message: Invalid argument: T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 in current term 1: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. W1010 20:58:45.182921 2450 leader_election.cc:275] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer b8062d66fd284035ae3bc2567bc11e7c: Timed out: RequestConsensusVote RPC to 127.6.150.4:44055 timed out after 1.511s I1010 20:58:45.182956 2450 leader_election.cc:248] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:45.183171 3934 raft_consensus.cc:1981] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.646s I1010 20:58:45.183192 3934 raft_consensus.cc:1823] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given W1010 20:58:45.193830 3390 leader_election.cc:275] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer b8062d66fd284035ae3bc2567bc11e7c: Timed out: RequestConsensusVote RPC to 127.6.150.4:44055 timed out after 1.525s W1010 20:58:45.193881 3390 leader_election.cc:275] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 7f623b00f5ae4b9c8e8940692d5dfd44: Timed out: RequestConsensusVote RPC to 127.6.150.7:45436 timed out after 1.525s I1010 20:58:45.193892 3390 leader_election.cc:248] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:45.194083 3935 raft_consensus.cc:1981] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.587s I1010 20:58:45.194100 3935 raft_consensus.cc:1823] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given W1010 20:58:45.195629 2919 leader_election.cc:275] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 1031512f5120422e9e22a0afabd78f61: Timed out: RequestConsensusVote RPC to 127.6.150.1:46963 timed out after 1.526s W1010 20:58:45.197695 2920 leader_election.cc:275] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 42e94fc3adbb419591836d222d946978: Timed out: RequestConsensusVote RPC to 127.6.150.8:35299 timed out after 1.526s I1010 20:58:45.197710 2920 leader_election.cc:248] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:45.197888 3936 raft_consensus.cc:1981] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.546s I1010 20:58:45.197916 3936 raft_consensus.cc:1823] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given W1010 20:58:45.233589 2684 leader_election.cc:275] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 42e94fc3adbb419591836d222d946978: Timed out: RequestConsensusVote RPC to 127.6.150.8:35299 timed out after 1.564s W1010 20:58:45.234122 2682 leader_election.cc:275] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 1380a27d6b824575a3a5e139bee139f4: Timed out: RequestConsensusVote RPC to 127.6.150.6:46774 timed out after 1.564s I1010 20:58:45.234134 2682 leader_election.cc:248] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:45.234278 3937 raft_consensus.cc:1981] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.623s I1010 20:58:45.234294 3937 raft_consensus.cc:1823] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given W1010 20:58:45.246172 2565 leader_election.cc:275] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 33ef4317e50042ba95e517992caa6531: Timed out: RequestConsensusVote RPC to 127.6.150.5:40764 timed out after 1.576s W1010 20:58:45.246306 2567 leader_election.cc:275] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer f6d15ccd64ec49be8396ed78e2a88377: Timed out: RequestConsensusVote RPC to 127.6.150.2:33554 timed out after 1.576s I1010 20:58:45.246333 2567 leader_election.cc:248] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:45.246525 3938 raft_consensus.cc:1981] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.509s I1010 20:58:45.246543 3938 raft_consensus.cc:1823] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given W1010 20:58:45.258355 3039 leader_election.cc:275] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer b8062d66fd284035ae3bc2567bc11e7c: Timed out: RequestConsensusVote RPC to 127.6.150.4:44055 timed out after 1.589s I1010 20:58:45.258373 3039 leader_election.cc:248] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:45.258559 3939 raft_consensus.cc:1981] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.530s I1010 20:58:45.258576 3939 raft_consensus.cc:1823] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given W1010 20:58:45.259341 3273 leader_election.cc:275] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer b8062d66fd284035ae3bc2567bc11e7c: Timed out: RequestConsensusVote RPC to 127.6.150.4:44055 timed out after 1.590s W1010 20:58:45.259361 3273 leader_election.cc:275] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 8de6cfa864d64c58bf4eafb689517c93: Timed out: RequestConsensusVote RPC to 127.6.150.3:40117 timed out after 1.590s I1010 20:58:45.259366 3273 leader_election.cc:248] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:45.259500 3940 raft_consensus.cc:1981] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.626s I1010 20:58:45.259516 3940 raft_consensus.cc:1823] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given W1010 20:58:45.288573 2802 leader_election.cc:275] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 84bde34331b94c4ea003e48f9317f792: Timed out: RequestConsensusVote RPC to 127.6.150.0:40113 timed out after 1.620s W1010 20:58:45.289275 2799 leader_election.cc:275] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 33ef4317e50042ba95e517992caa6531: Timed out: RequestConsensusVote RPC to 127.6.150.5:40764 timed out after 1.620s I1010 20:58:45.289305 2799 leader_election.cc:248] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:45.289485 3941 raft_consensus.cc:1981] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.627s I1010 20:58:45.289501 3941 raft_consensus.cc:1823] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given W1010 20:58:45.311717 3156 leader_election.cc:275] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 84bde34331b94c4ea003e48f9317f792: Timed out: RequestConsensusVote RPC to 127.6.150.0:40113 timed out after 1.643s W1010 20:58:45.312273 3153 leader_election.cc:275] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 33ef4317e50042ba95e517992caa6531: Timed out: RequestConsensusVote RPC to 127.6.150.5:40764 timed out after 1.643s I1010 20:58:45.312285 3153 leader_election.cc:248] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:45.312420 3942 raft_consensus.cc:1981] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.510s I1010 20:58:45.312436 3942 raft_consensus.cc:1823] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given W1010 20:58:45.316730 3507 leader_election.cc:275] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 7f623b00f5ae4b9c8e8940692d5dfd44: Timed out: RequestConsensusVote RPC to 127.6.150.7:45436 timed out after 1.648s W1010 20:58:45.316804 3504 leader_election.cc:275] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 33ef4317e50042ba95e517992caa6531: Timed out: RequestConsensusVote RPC to 127.6.150.5:40764 timed out after 1.648s I1010 20:58:45.316813 3504 leader_election.cc:248] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:45.316961 3943 raft_consensus.cc:1981] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.502s I1010 20:58:45.316975 3943 raft_consensus.cc:1823] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given W1010 20:58:45.364399 3673 consensus_meta.cc:188] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6: Time spent flushing consensus metadata: real 1.699s user 0.000s sys 0.000s W1010 20:58:45.364399 3891 consensus_meta.cc:188] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 1.699s user 0.000s sys 0.000s W1010 20:58:45.364399 3754 consensus_meta.cc:188] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 1.699s user 0.000s sys 0.000s W1010 20:58:45.364421 3829 consensus_meta.cc:188] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 1.698s user 0.000s sys 0.000s W1010 20:58:45.364480 3715 consensus_meta.cc:188] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 1.699s user 0.000s sys 0.000s W1010 20:58:45.364554 3678 consensus_meta.cc:188] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 1.697s user 0.000s sys 0.000s W1010 20:58:45.364719 3870 consensus_meta.cc:188] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 2.608s user 0.000s sys 0.000s I1010 20:58:45.364735 3870 raft_consensus.cc:1981] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.524s W1010 20:58:45.364761 3794 consensus_meta.cc:188] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 2.910s user 0.000s sys 0.000s I1010 20:58:45.364766 3870 raft_consensus.cc:391] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:58:45.364796 3794 raft_consensus.cc:1981] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.553s W1010 20:58:45.364821 3722 consensus_meta.cc:188] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6: Time spent flushing consensus metadata: real 2.908s user 0.000s sys 0.000s I1010 20:58:45.364825 3794 raft_consensus.cc:391] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } I1010 20:58:45.364836 3722 raft_consensus.cc:1981] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.553s I1010 20:58:45.364868 3722 raft_consensus.cc:391] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } W1010 20:58:45.364925 3881 consensus_meta.cc:188] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 2.923s user 0.000s sys 0.000s I1010 20:58:45.364969 3881 raft_consensus.cc:1981] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.515s I1010 20:58:45.364997 3881 raft_consensus.cc:391] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } W1010 20:58:45.365013 3848 consensus_meta.cc:188] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 1.698s user 0.000s sys 0.000s W1010 20:58:45.365151 3779 consensus_meta.cc:188] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 3.015s user 0.000s sys 0.000s W1010 20:58:45.365152 3763 consensus_meta.cc:188] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 1.699s user 0.000s sys 0.000s I1010 20:58:45.365170 3779 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.639s I1010 20:58:45.365195 3779 raft_consensus.cc:391] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } W1010 20:58:45.365262 3837 consensus_meta.cc:188] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 2.728s user 0.000s sys 0.000s W1010 20:58:45.365278 3667 consensus_meta.cc:188] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 2.848s user 0.000s sys 0.000s I1010 20:58:45.365289 3837 raft_consensus.cc:1981] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.505s I1010 20:58:45.365299 3667 raft_consensus.cc:1981] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.603s I1010 20:58:45.365325 3837 raft_consensus.cc:391] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } I1010 20:58:45.365330 3667 raft_consensus.cc:391] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } W1010 20:58:45.365521 3698 consensus_meta.cc:188] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 2.565s user 0.000s sys 0.000s W1010 20:58:45.365525 3697 consensus_meta.cc:188] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 1.699s user 0.000s sys 0.000s I1010 20:58:45.365552 3698 raft_consensus.cc:1981] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.566s I1010 20:58:45.365581 3698 raft_consensus.cc:391] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } W1010 20:58:45.365610 3816 consensus_meta.cc:188] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 2.250s user 0.000s sys 0.000s I1010 20:58:45.365629 3816 raft_consensus.cc:1981] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.555s I1010 20:58:45.365656 3816 raft_consensus.cc:391] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } W1010 20:58:45.365692 3713 consensus_meta.cc:188] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 2.512s user 0.000s sys 0.000s I1010 20:58:45.365707 3713 raft_consensus.cc:1981] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.572s I1010 20:58:45.365726 3713 raft_consensus.cc:391] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } W1010 20:58:45.365761 3671 consensus_meta.cc:188] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 2.821s user 0.000s sys 0.000s I1010 20:58:45.365777 3671 raft_consensus.cc:1981] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.637s I1010 20:58:45.365808 3671 raft_consensus.cc:391] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } W1010 20:58:45.365830 3882 consensus_meta.cc:188] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.699s user 0.000s sys 0.000s W1010 20:58:45.366011 3753 consensus_meta.cc:188] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 2.819s user 0.000s sys 0.000s I1010 20:58:45.366024 3753 raft_consensus.cc:1981] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.600s I1010 20:58:45.366041 3753 raft_consensus.cc:391] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } W1010 20:58:45.366153 3869 consensus_meta.cc:188] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.701s user 0.000s sys 0.000s W1010 20:58:45.366266 3898 consensus_meta.cc:188] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 2.124s user 0.000s sys 0.000s I1010 20:58:45.366284 3898 raft_consensus.cc:1981] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.623s I1010 20:58:45.366302 3898 raft_consensus.cc:391] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } W1010 20:58:45.366336 3720 consensus_meta.cc:188] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.701s user 0.000s sys 0.000s I1010 20:58:45.366353 3715 leader_election.cc:223] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 1 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 I1010 20:58:45.366369 3715 leader_election.cc:223] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 1 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 I1010 20:58:45.366400 3678 leader_election.cc:223] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 1 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 W1010 20:58:45.366406 3805 consensus_meta.cc:188] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.699s user 0.000s sys 0.000s I1010 20:58:45.366412 3678 leader_election.cc:223] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 1 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 I1010 20:58:45.366441 3829 leader_election.cc:223] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: Requesting vote from peer 580cbaf7092444dfbc6fb344da88d7b6 I1010 20:58:45.366459 3829 leader_election.cc:223] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 I1010 20:58:45.366493 3673 leader_election.cc:223] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 1 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 I1010 20:58:45.366506 3673 leader_election.cc:223] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 1 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 I1010 20:58:45.366650 3754 leader_election.cc:223] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 1 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:58:45.366703 3754 leader_election.cc:223] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 1 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 I1010 20:58:45.366704 2513 raft_consensus.cc:1609] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 in current term 1: Already voted for candidate 84bde34331b94c4ea003e48f9317f792 in this term. W1010 20:58:45.366785 2513 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.6:59570 (request call id 1) took 1692ms (client timeout 1643). W1010 20:58:45.366850 2513 rpcz_store.cc:238] Trace: 1010 20:58:43.674191 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:43.674205 (+ 14us) service_pool.cc:202] Handling call 1010 20:58:45.366774 (+1692569us) spinlock_profiling.cc:255] Waited 1.69 s on lock 0x409beb0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007ff4d62cf181 00007ff4d51b747c ffffffffffffffff 1010 20:58:45.366779 (+ 5us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3892605312} I1010 20:58:45.366869 3101 raft_consensus.cc:1609] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 in current term 1: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. W1010 20:58:45.366948 3101 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.6:59112 (request call id 1) took 1693ms (client timeout 1643). W1010 20:58:45.366986 3101 rpcz_store.cc:238] Trace: 1010 20:58:43.673705 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:43.673715 (+ 10us) service_pool.cc:202] Handling call 1010 20:58:45.366939 (+1693224us) spinlock_profiling.cc:255] Waited 1.69 s on lock 0x4d756d0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007fc8aa54b181 00007fc8a943347c ffffffffffffffff 1010 20:58:45.366943 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3894209408} I1010 20:58:45.367039 3101 raft_consensus.cc:1609] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 in current term 1: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. I1010 20:58:45.367126 3848 leader_election.cc:223] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 1 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:58:45.367136 2447 leader_election.cc:361] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 1 election: Vote denied by peer 33ef4317e50042ba95e517992caa6531. Message: Invalid argument: T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 in current term 1: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. I1010 20:58:45.367142 3848 leader_election.cc:223] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 1 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:58:45.367213 3697 leader_election.cc:223] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 I1010 20:58:45.367218 2513 raft_consensus.cc:1609] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 1: Already voted for candidate 84bde34331b94c4ea003e48f9317f792 in this term. I1010 20:58:45.367230 3697 leader_election.cc:223] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:58:45.367265 3763 leader_election.cc:223] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 1 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 I1010 20:58:45.367278 3763 leader_election.cc:223] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 1 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:58:45.367288 3039 leader_election.cc:361] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 1 election: Vote denied by peer 84bde34331b94c4ea003e48f9317f792. Message: Invalid argument: T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 1: Already voted for candidate 84bde34331b94c4ea003e48f9317f792 in this term. I1010 20:58:45.367318 3891 leader_election.cc:223] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 1 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:58:45.367331 3891 leader_election.cc:223] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 1 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 I1010 20:58:45.367373 3454 raft_consensus.cc:1609] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 1: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. W1010 20:58:45.367429 3454 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.2:40678 (request call id 1) took 1694ms (client timeout 1564). W1010 20:58:45.367449 3454 rpcz_store.cc:238] Trace: 1010 20:58:43.673032 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:43.673043 (+ 11us) service_pool.cc:202] Handling call 1010 20:58:45.367422 (+1694379us) spinlock_profiling.cc:255] Waited 1.69 s on lock 0x420c470. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007fd760afa181 00007fd75f9e247c ffffffffffffffff 1010 20:58:45.367426 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3896910208} I1010 20:58:45.367825 3882 leader_election.cc:223] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:58:45.367851 3882 leader_election.cc:223] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 I1010 20:58:45.367943 3869 leader_election.cc:223] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 I1010 20:58:45.367954 3869 leader_election.cc:223] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:58:45.368024 3720 leader_election.cc:223] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:58:45.368036 3219 raft_consensus.cc:1609] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 in current term 1: Already voted for candidate 1380a27d6b824575a3a5e139bee139f4 in this term. I1010 20:58:45.368072 2984 raft_consensus.cc:1609] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 1: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:58:45.368116 2447 leader_election.cc:361] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 1 election: Vote denied by peer 1380a27d6b824575a3a5e139bee139f4. Message: Invalid argument: T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 in current term 1: Already voted for candidate 1380a27d6b824575a3a5e139bee139f4 in this term. I1010 20:58:45.368120 3805 leader_election.cc:223] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 I1010 20:58:45.368125 2447 leader_election.cc:248] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:45.368137 3805 leader_election.cc:223] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:58:45.368165 2983 raft_consensus.cc:1654] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 for term 1 because replica is already servicing an update from a current leader or another vote. W1010 20:58:45.368190 2983 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.0:43796 (request call id 1) took 1693ms (client timeout 1511). W1010 20:58:45.368217 2983 rpcz_store.cc:238] Trace: 1010 20:58:43.674584 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:43.674597 (+ 13us) service_pool.cc:202] Handling call 1010 20:58:45.368181 (+1693584us) spinlock_profiling.cc:255] Waited 1.69 s on lock 0x4dbe470. stack: 1010 20:58:45.368184 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3895011072} W1010 20:58:45.368290 2984 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.5:55772 (request call id 1) took 1695ms (client timeout 1589). W1010 20:58:45.368312 2984 rpcz_store.cc:238] Trace: 1010 20:58:43.673024 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:43.673033 (+ 9us) service_pool.cc:202] Handling call 1010 20:58:45.368282 (+1695249us) spinlock_profiling.cc:255] Waited 1.69 s on lock 0x4dbe470. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007fe1aafaf181 00007fe1a9e9747c ffffffffffffffff 1010 20:58:45.368285 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3898180352} I1010 20:58:45.368331 3720 leader_election.cc:223] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 I1010 20:58:45.368355 2985 raft_consensus.cc:1609] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 1: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. W1010 20:58:45.368387 2985 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.8:46146 (request call id 1) took 1696ms (client timeout 1525). W1010 20:58:45.368408 2985 rpcz_store.cc:238] Trace: 1010 20:58:43.671940 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:43.671955 (+ 15us) service_pool.cc:202] Handling call 1010 20:58:45.368379 (+1696424us) spinlock_profiling.cc:255] Waited 1.7 s on lock 0x4a3b970. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007fe1aafaf181 00007fe1a9e9747c ffffffffffffffff 1010 20:58:45.368382 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3901085056} I1010 20:58:45.368443 2986 raft_consensus.cc:1609] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 1: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. W1010 20:58:45.368486 2986 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.7:55183 (request call id 1) took 1696ms (client timeout 1590). W1010 20:58:45.368521 2986 rpcz_store.cc:238] Trace: 1010 20:58:43.671773 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:43.671785 (+ 12us) service_pool.cc:202] Handling call 1010 20:58:45.368470 (+1696685us) spinlock_profiling.cc:255] Waited 1.7 s on lock 0x4dbe1d0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007fe1aafaf181 00007fe1a9e9747c ffffffffffffffff 1010 20:58:45.368481 (+ 11us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3901272448} I1010 20:58:45.368687 3454 raft_consensus.cc:1609] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 1: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:58:45.368850 2920 leader_election.cc:361] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: Vote denied by peer 42e94fc3adbb419591836d222d946978. Message: Invalid argument: T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 1: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:58:45.369475 3219 raft_consensus.cc:1609] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 1: Already voted for candidate 1380a27d6b824575a3a5e139bee139f4 in this term. I1010 20:58:45.369565 2682 leader_election.cc:361] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 1 election: Vote denied by peer 1380a27d6b824575a3a5e139bee139f4. Message: Invalid argument: T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 1: Already voted for candidate 1380a27d6b824575a3a5e139bee139f4 in this term. I1010 20:58:45.369638 2513 raft_consensus.cc:1609] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 1: Already voted for candidate 84bde34331b94c4ea003e48f9317f792 in this term. I1010 20:58:45.369704 2921 leader_election.cc:361] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: Vote denied by peer 84bde34331b94c4ea003e48f9317f792. Message: Invalid argument: T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 1: Already voted for candidate 84bde34331b94c4ea003e48f9317f792 in this term. I1010 20:58:45.369998 3219 raft_consensus.cc:1609] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 1: Already voted for candidate 1380a27d6b824575a3a5e139bee139f4 in this term. I1010 20:58:45.370100 3036 leader_election.cc:361] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 1 election: Vote denied by peer 1380a27d6b824575a3a5e139bee139f4. Message: Invalid argument: T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 1: Already voted for candidate 1380a27d6b824575a3a5e139bee139f4 in this term. I1010 20:58:45.370111 3036 leader_election.cc:248] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:45.370138 3956 raft_consensus.cc:1981] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.648s I1010 20:58:45.370148 3956 raft_consensus.cc:1823] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given I1010 20:58:45.370687 2748 raft_consensus.cc:1609] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 1: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:58:45.370746 3389 leader_election.cc:361] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 1 election: Vote denied by peer f6d15ccd64ec49be8396ed78e2a88377. Message: Invalid argument: T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 1: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:58:45.370816 2748 raft_consensus.cc:1609] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 in current term 1: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:58:45.370877 3155 leader_election.cc:361] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 1 election: Vote denied by peer f6d15ccd64ec49be8396ed78e2a88377. Message: Invalid argument: T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 in current term 1: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:58:45.370970 2864 raft_consensus.cc:1609] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 1: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. I1010 20:58:45.370982 3101 raft_consensus.cc:1609] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 1: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. I1010 20:58:45.371023 2921 leader_election.cc:361] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: Vote denied by peer 8de6cfa864d64c58bf4eafb689517c93. Message: Invalid argument: T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 1: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. I1010 20:58:45.371026 3971 raft_consensus.cc:1981] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.634s I1010 20:58:45.371048 3971 raft_consensus.cc:1823] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given I1010 20:58:45.371064 2918 leader_election.cc:361] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: Vote denied by peer 33ef4317e50042ba95e517992caa6531. Message: Invalid argument: T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 1: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. I1010 20:58:45.371089 2918 leader_election.cc:248] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:45.371068 2986 raft_consensus.cc:1609] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 1: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:58:45.371212 2802 leader_election.cc:361] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c. Message: Invalid argument: T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 1: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:58:45.371453 3974 raft_consensus.cc:1981] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.577s I1010 20:58:45.371621 3335 raft_consensus.cc:1609] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 in current term 1: Already voted for candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in this term. I1010 20:58:45.371729 3507 leader_election.cc:361] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 1 election: Vote denied by peer 7f623b00f5ae4b9c8e8940692d5dfd44. Message: Invalid argument: T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 in current term 1: Already voted for candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in this term. I1010 20:58:45.371747 3974 raft_consensus.cc:1823] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given I1010 20:58:45.372094 3571 raft_consensus.cc:1609] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 1: Already voted for candidate 580cbaf7092444dfbc6fb344da88d7b6 in this term. I1010 20:58:45.372171 3273 leader_election.cc:361] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: Vote denied by peer 580cbaf7092444dfbc6fb344da88d7b6. Message: Invalid argument: T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 1: Already voted for candidate 580cbaf7092444dfbc6fb344da88d7b6 in this term. I1010 20:58:45.374367 3334 raft_consensus.cc:1609] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 1: Already voted for candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in this term. I1010 20:58:45.374428 2921 leader_election.cc:361] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: Vote denied by peer 7f623b00f5ae4b9c8e8940692d5dfd44. Message: Invalid argument: T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 1: Already voted for candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in this term. I1010 20:58:46.716588 3763 raft_consensus.cc:375] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:46.716608 3763 raft_consensus.cc:2031] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Advancing to term 2 W1010 20:58:46.881795 3387 leader_election.cc:275] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 1380a27d6b824575a3a5e139bee139f4: Timed out: RequestConsensusVote RPC to 127.6.150.6:46774 timed out after 1.512s I1010 20:58:46.881813 3387 leader_election.cc:248] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:46.881968 3976 raft_consensus.cc:1981] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.572s I1010 20:58:46.881985 3976 raft_consensus.cc:1823] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given W1010 20:58:46.885828 2799 leader_election.cc:275] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 1380a27d6b824575a3a5e139bee139f4: Timed out: RequestConsensusVote RPC to 127.6.150.6:46774 timed out after 1.517s I1010 20:58:46.885843 2799 leader_election.cc:248] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:46.885964 3977 raft_consensus.cc:1981] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.524s I1010 20:58:46.885979 3977 raft_consensus.cc:1823] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given W1010 20:58:46.904705 3271 leader_election.cc:275] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 1031512f5120422e9e22a0afabd78f61: Timed out: RequestConsensusVote RPC to 127.6.150.1:46963 timed out after 1.537s I1010 20:58:46.904723 3271 leader_election.cc:248] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:46.904881 3978 raft_consensus.cc:1981] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.575s I1010 20:58:46.904896 3978 raft_consensus.cc:1823] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given W1010 20:58:46.907202 2684 leader_election.cc:275] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 42e94fc3adbb419591836d222d946978: Timed out: RequestConsensusVote RPC to 127.6.150.8:35299 timed out after 1.538s I1010 20:58:46.907230 2684 leader_election.cc:248] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:46.909351 3869 raft_consensus.cc:375] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:46.909364 3869 raft_consensus.cc:2031] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Advancing to term 2 W1010 20:58:46.927217 2568 leader_election.cc:275] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 7f623b00f5ae4b9c8e8940692d5dfd44: Timed out: RequestConsensusVote RPC to 127.6.150.7:45436 timed out after 1.559s W1010 20:58:46.927258 2568 leader_election.cc:275] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 84bde34331b94c4ea003e48f9317f792: Timed out: RequestConsensusVote RPC to 127.6.150.0:40113 timed out after 1.559s I1010 20:58:46.927263 2568 leader_election.cc:248] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:46.927464 3980 raft_consensus.cc:1981] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.643s I1010 20:58:46.927481 3980 raft_consensus.cc:1823] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given W1010 20:58:46.927980 2921 leader_election.cc:275] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 7f623b00f5ae4b9c8e8940692d5dfd44: Timed out: RequestConsensusVote RPC to 127.6.150.7:45436 timed out after 1.558s I1010 20:58:46.927992 2921 leader_election.cc:248] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:46.928124 3981 raft_consensus.cc:1981] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.540s I1010 20:58:46.928138 3981 raft_consensus.cc:1823] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given I1010 20:58:46.931493 3848 raft_consensus.cc:375] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:46.931506 3848 raft_consensus.cc:2031] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Advancing to term 2 W1010 20:58:46.940439 2921 leader_election.cc:275] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 8de6cfa864d64c58bf4eafb689517c93: Timed out: RequestConsensusVote RPC to 127.6.150.3:40117 timed out after 1.571s I1010 20:58:46.940450 2921 leader_election.cc:248] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. W1010 20:58:46.947216 3870 consensus_meta.cc:188] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 1.582s user 0.000s sys 0.000s W1010 20:58:46.947250 3794 consensus_meta.cc:188] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 1.582s user 0.000s sys 0.000s I1010 20:58:46.947405 3870 leader_election.cc:223] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:58:46.947407 3794 leader_election.cc:223] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 1 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 I1010 20:58:46.947423 3870 leader_election.cc:223] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:58:46.947425 3794 leader_election.cc:223] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 1 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 I1010 20:58:46.947537 2986 raft_consensus.cc:1609] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 1: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. W1010 20:58:46.947579 3889 consensus_meta.cc:188] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 3.817s user 0.000s sys 0.000s I1010 20:58:46.947598 3889 raft_consensus.cc:1981] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.545s I1010 20:58:46.947630 3889 raft_consensus.cc:391] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:58:46.947643 3273 leader_election.cc:361] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c. Message: Invalid argument: T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 1: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. W1010 20:58:46.947657 3692 consensus_meta.cc:188] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 4.422s user 0.000s sys 0.000s I1010 20:58:46.947674 3692 raft_consensus.cc:1981] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.534s I1010 20:58:46.947674 3336 raft_consensus.cc:1609] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 1: Already voted for candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in this term. I1010 20:58:46.947695 3692 raft_consensus.cc:391] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } W1010 20:58:46.947749 3818 consensus_meta.cc:188] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 3.790s user 0.000s sys 0.000s W1010 20:58:46.947773 3336 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.8:39867 (request call id 1) took 3274ms (client timeout 1525). I1010 20:58:46.947790 3818 raft_consensus.cc:1981] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.552s W1010 20:58:46.947816 3336 rpcz_store.cc:238] Trace: 1010 20:58:43.673496 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:43.673506 (+ 10us) service_pool.cc:202] Handling call 1010 20:58:46.947764 (+3274258us) spinlock_profiling.cc:255] Waited 3.27 s on lock 0x42161d0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f3fb2fbf181 00007f3fb1ea747c ffffffffffffffff 1010 20:58:46.947768 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7530544384} I1010 20:58:46.947841 3818 raft_consensus.cc:391] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:58:46.948091 3333 raft_consensus.cc:1654] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c for term 1 because replica is already servicing an update from a current leader or another vote. W1010 20:58:46.948146 3333 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.4:46735 (request call id 2) took 1573ms (client timeout 1558). W1010 20:58:46.948169 3333 rpcz_store.cc:238] Trace: 1010 20:58:45.374324 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:45.374344 (+ 20us) service_pool.cc:202] Handling call 1010 20:58:46.948139 (+1573795us) spinlock_profiling.cc:255] Waited 1.57 s on lock 0x42161d0. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f3fb2fbf181 00007f3fb1ea747c ffffffffffffffff 1010 20:58:46.948143 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3619568512} W1010 20:58:46.948173 3722 consensus_meta.cc:188] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6: Time spent flushing consensus metadata: real 1.583s user 0.000s sys 0.000s I1010 20:58:46.948338 3722 leader_election.cc:223] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 1 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:58:46.948379 3722 leader_election.cc:223] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 1 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 W1010 20:58:46.948622 3694 consensus_meta.cc:188] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6: Time spent flushing consensus metadata: real 4.444s user 0.000s sys 0.000s I1010 20:58:46.948657 3694 raft_consensus.cc:1981] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.585s I1010 20:58:46.948678 3694 raft_consensus.cc:391] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:58:46.948956 3451 raft_consensus.cc:1609] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 1: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:58:46.949059 3272 leader_election.cc:361] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: Vote denied by peer 42e94fc3adbb419591836d222d946978. Message: Invalid argument: T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 1: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:58:46.949069 3272 leader_election.cc:248] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:46.949142 2513 raft_consensus.cc:1609] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 1: Already voted for candidate 84bde34331b94c4ea003e48f9317f792 in this term. W1010 20:58:46.949187 2513 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.1:38540 (request call id 1) took 1576ms (client timeout 1559). I1010 20:58:46.949196 3987 raft_consensus.cc:1981] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.513s W1010 20:58:46.949214 2513 rpcz_store.cc:238] Trace: 1010 20:58:45.372424 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:45.372435 (+ 11us) service_pool.cc:202] Handling call 1010 20:58:46.949179 (+1576744us) spinlock_profiling.cc:255] Waited 1.58 s on lock 0x444c1d0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007ff4d62cf181 00007ff4d51b747c ffffffffffffffff 1010 20:58:46.949183 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3626397568} I1010 20:58:46.949236 3987 raft_consensus.cc:1823] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given I1010 20:58:46.949946 2629 raft_consensus.cc:1609] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 in current term 1: Already voted for candidate 1031512f5120422e9e22a0afabd78f61 in this term. I1010 20:58:46.950067 2448 leader_election.cc:361] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 1 election: Vote denied by peer 1031512f5120422e9e22a0afabd78f61. Message: Invalid argument: T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 in current term 1: Already voted for candidate 1031512f5120422e9e22a0afabd78f61 in this term. W1010 20:58:46.951481 3837 consensus_meta.cc:188] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 1.586s user 0.000s sys 0.000s I1010 20:58:46.951613 3837 leader_election.cc:223] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 1 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:58:46.951637 3837 leader_election.cc:223] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 1 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 W1010 20:58:46.951688 3789 consensus_meta.cc:188] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 3.285s user 0.000s sys 0.000s I1010 20:58:46.951829 3789 leader_election.cc:223] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 1 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:58:46.951843 3789 leader_election.cc:223] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 1 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 W1010 20:58:46.951864 3859 consensus_meta.cc:188] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 4.174s user 0.000s sys 0.000s I1010 20:58:46.951910 3217 raft_consensus.cc:1654] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 for term 1 because replica is already servicing an update from a current leader or another vote. I1010 20:58:46.951923 3859 raft_consensus.cc:1981] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.526s I1010 20:58:46.951977 3859 raft_consensus.cc:391] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } I1010 20:58:46.951992 3848 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.714s I1010 20:58:46.952018 3848 raft_consensus.cc:391] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } W1010 20:58:46.952020 3217 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.3:52871 (request call id 1) took 1580ms (client timeout 1517). W1010 20:58:46.952021 3667 consensus_meta.cc:188] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 1.587s user 0.000s sys 0.000s W1010 20:58:46.952045 3217 rpcz_store.cc:238] Trace: 1010 20:58:45.371496 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:45.371504 (+ 8us) service_pool.cc:202] Handling call 1010 20:58:46.952011 (+1580507us) spinlock_profiling.cc:255] Waited 1.58 s on lock 0x3b756d0. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f8ce19fe181 00007f8ce08e647c ffffffffffffffff 1010 20:58:46.952015 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3634874752,"tcmalloc_contention_cycles":3968} I1010 20:58:46.952107 3218 raft_consensus.cc:1609] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 1: Already voted for candidate 1380a27d6b824575a3a5e139bee139f4 in this term. I1010 20:58:46.952111 3667 leader_election.cc:223] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 1 election: Requesting vote from peer 580cbaf7092444dfbc6fb344da88d7b6 I1010 20:58:46.952126 3667 leader_election.cc:223] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 1 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 W1010 20:58:46.952142 3218 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.4:46303 (request call id 1) took 1580ms (client timeout 1608). W1010 20:58:46.952162 3218 rpcz_store.cc:238] Trace: 1010 20:58:45.371321 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:45.371343 (+ 22us) service_pool.cc:202] Handling call 1010 20:58:46.952134 (+1580791us) spinlock_profiling.cc:255] Waited 1.58 s on lock 0x3b756d0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f8ce19fe181 00007f8ce08e647c ffffffffffffffff 1010 20:58:46.952137 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3635719936} I1010 20:58:46.952288 2918 leader_election.cc:361] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: Vote denied by peer 1380a27d6b824575a3a5e139bee139f4. Message: Invalid argument: T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 1: Already voted for candidate 1380a27d6b824575a3a5e139bee139f4 in this term. I1010 20:58:46.952311 2918 leader_election.cc:248] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. W1010 20:58:46.952378 3867 consensus_meta.cc:188] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 4.430s user 0.000s sys 0.000s I1010 20:58:46.952404 3867 raft_consensus.cc:1981] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.604s I1010 20:58:46.952425 3867 raft_consensus.cc:391] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } I1010 20:58:46.952474 3994 raft_consensus.cc:1981] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.553s I1010 20:58:46.952493 3994 raft_consensus.cc:1823] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given W1010 20:58:46.952533 3698 consensus_meta.cc:188] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 1.587s user 0.000s sys 0.000s I1010 20:58:46.952659 3698 leader_election.cc:223] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 1 election: Requesting vote from peer 580cbaf7092444dfbc6fb344da88d7b6 I1010 20:58:46.952679 3698 leader_election.cc:223] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 1 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:58:46.952695 2748 raft_consensus.cc:1609] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 in current term 1: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:58:46.952800 3673 raft_consensus.cc:375] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:46.952811 3673 raft_consensus.cc:2031] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Advancing to term 2 W1010 20:58:46.952852 3729 consensus_meta.cc:188] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 4.043s user 0.000s sys 0.000s I1010 20:58:46.952869 3729 raft_consensus.cc:1981] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.520s I1010 20:58:46.952891 3729 raft_consensus.cc:391] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } W1010 20:58:46.952909 3819 consensus_meta.cc:188] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 3.999s user 0.000s sys 0.000s I1010 20:58:46.952929 3819 raft_consensus.cc:1981] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.623s I1010 20:58:46.952950 3819 raft_consensus.cc:391] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } I1010 20:58:46.953065 2631 raft_consensus.cc:1609] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 1: Already voted for candidate 1031512f5120422e9e22a0afabd78f61 in this term. I1010 20:58:46.953126 3506 leader_election.cc:361] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 1 election: Vote denied by peer f6d15ccd64ec49be8396ed78e2a88377. Message: Invalid argument: T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 in current term 1: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. W1010 20:58:46.953157 2631 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.7:33193 (request call id 1) took 1581ms (client timeout 1537). W1010 20:58:46.953181 2631 rpcz_store.cc:238] Trace: 1010 20:58:45.371944 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:45.371953 (+ 9us) service_pool.cc:202] Handling call 1010 20:58:46.953149 (+1581196us) spinlock_profiling.cc:255] Waited 1.58 s on lock 0x3f856d0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f324f2ae181 00007f324e19647c ffffffffffffffff 1010 20:58:46.953153 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3636508544} I1010 20:58:46.953238 2630 raft_consensus.cc:1654] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 for term 1 because replica is already servicing an update from a current leader or another vote. W1010 20:58:46.953272 2630 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.9:55298 (request call id 1) took 1581ms (client timeout 1593). W1010 20:58:46.953292 2630 rpcz_store.cc:238] Trace: 1010 20:58:45.371969 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:45.371977 (+ 8us) service_pool.cc:202] Handling call 1010 20:58:46.953267 (+1581290us) spinlock_profiling.cc:255] Waited 1.58 s on lock 0x3f856d0. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f324f2ae181 00007f324e19647c ffffffffffffffff 1010 20:58:46.953269 (+ 2us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3636851072} I1010 20:58:46.953357 3505 leader_election.cc:361] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 1 election: Vote denied by peer 1031512f5120422e9e22a0afabd78f61. Message: Service unavailable: T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 for term 1 because replica is already servicing an update from a current leader or another vote. I1010 20:58:46.953367 3505 leader_election.cc:248] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:46.953596 3336 raft_consensus.cc:1609] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 1: Already voted for candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in this term. I1010 20:58:46.953655 3571 raft_consensus.cc:1609] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 1: Already voted for candidate 580cbaf7092444dfbc6fb344da88d7b6 in this term. I1010 20:58:46.953661 2568 leader_election.cc:361] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 1 election: Vote denied by peer 7f623b00f5ae4b9c8e8940692d5dfd44. Message: Invalid argument: T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 1: Already voted for candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in this term. I1010 20:58:46.953716 2685 leader_election.cc:361] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 1 election: Vote denied by peer 580cbaf7092444dfbc6fb344da88d7b6. Message: Invalid argument: T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 1: Already voted for candidate 580cbaf7092444dfbc6fb344da88d7b6 in this term. I1010 20:58:46.954556 2863 raft_consensus.cc:1609] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 in current term 1: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. I1010 20:58:46.954625 3156 leader_election.cc:361] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 1 election: Vote denied by peer 8de6cfa864d64c58bf4eafb689517c93. Message: Invalid argument: T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 in current term 1: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. W1010 20:58:46.954692 3671 consensus_meta.cc:188] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 1.589s user 0.000s sys 0.000s W1010 20:58:46.954715 3816 consensus_meta.cc:188] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 1.589s user 0.000s sys 0.000s I1010 20:58:46.954727 2984 raft_consensus.cc:1609] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 in current term 1: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:58:46.954787 3156 leader_election.cc:361] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 1 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c. Message: Invalid argument: T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 in current term 1: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:58:46.954795 3156 leader_election.cc:248] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:46.954805 3671 leader_election.cc:223] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 W1010 20:58:46.954803 3713 consensus_meta.cc:188] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 1.589s user 0.000s sys 0.000s I1010 20:58:46.954812 3816 leader_election.cc:223] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 1 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:58:46.954826 3816 leader_election.cc:223] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 1 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 I1010 20:58:46.954829 3671 leader_election.cc:223] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 W1010 20:58:46.954872 3898 consensus_meta.cc:188] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.589s user 0.000s sys 0.000s I1010 20:58:46.954900 3713 leader_election.cc:223] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 1 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:58:46.954915 3713 leader_election.cc:223] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 1 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 W1010 20:58:46.954968 3894 consensus_meta.cc:188] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 3.288s user 0.000s sys 0.000s I1010 20:58:46.954994 3898 leader_election.cc:223] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: Requesting vote from peer 580cbaf7092444dfbc6fb344da88d7b6 I1010 20:58:46.955010 3898 leader_election.cc:223] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 I1010 20:58:46.955055 3894 leader_election.cc:223] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: Requesting vote from peer 580cbaf7092444dfbc6fb344da88d7b6 I1010 20:58:46.955066 3894 leader_election.cc:223] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 W1010 20:58:46.955143 3817 consensus_meta.cc:188] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 4.211s user 0.000s sys 0.000s I1010 20:58:46.955158 3817 raft_consensus.cc:1981] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.511s I1010 20:58:46.955178 3817 raft_consensus.cc:391] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } W1010 20:58:46.955176 3857 consensus_meta.cc:188] T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 4.088s user 0.000s sys 0.000s I1010 20:58:46.955193 3857 raft_consensus.cc:1981] T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.598s I1010 20:58:46.955214 3857 raft_consensus.cc:391] T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } W1010 20:58:46.955230 3871 consensus_meta.cc:188] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 3.587s user 0.000s sys 0.000s I1010 20:58:46.955247 3871 raft_consensus.cc:1981] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.607s I1010 20:58:46.955268 3871 raft_consensus.cc:391] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:58:46.955276 3869 raft_consensus.cc:1981] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.675s I1010 20:58:46.955299 3869 raft_consensus.cc:391] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:58:46.955377 3102 raft_consensus.cc:1609] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 1: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. W1010 20:58:46.955418 3102 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.1:45293 (request call id 1) took 3281ms (client timeout 1576). W1010 20:58:46.955436 3102 rpcz_store.cc:238] Trace: 1010 20:58:43.673610 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:43.673658 (+ 48us) service_pool.cc:202] Handling call 1010 20:58:46.955410 (+3281752us) spinlock_profiling.cc:255] Waited 3.28 s on lock 0x4d75970. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007fc8aa54b181 00007fc8a943347c ffffffffffffffff 1010 20:58:46.955414 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7547919616} I1010 20:58:46.955435 3453 raft_consensus.cc:1609] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 1: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:58:46.955454 2986 raft_consensus.cc:1609] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 in current term 1: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. W1010 20:58:46.955473 3453 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.4:56561 (request call id 1) took 3282ms (client timeout 1526). W1010 20:58:46.955492 3453 rpcz_store.cc:238] Trace: 1010 20:58:43.673186 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:43.673204 (+ 18us) service_pool.cc:202] Handling call 1010 20:58:46.955465 (+3282261us) spinlock_profiling.cc:255] Waited 3.28 s on lock 0x3e4fc10. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007fd760afa181 00007fd75f9e247c ffffffffffffffff 1010 20:58:46.955469 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7549084928} I1010 20:58:46.955571 2985 raft_consensus.cc:1654] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 for term 1 because replica is already servicing an update from a current leader or another vote. I1010 20:58:46.955662 2685 leader_election.cc:361] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 1 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c. Message: Service unavailable: T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 for term 1 because replica is already servicing an update from a current leader or another vote. I1010 20:58:46.955670 2685 leader_election.cc:248] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:46.955723 3507 leader_election.cc:361] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 1 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c. Message: Invalid argument: T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 in current term 1: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:58:46.955734 3507 leader_election.cc:248] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:46.955765 2986 raft_consensus.cc:1609] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 1: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:58:46.955848 3390 leader_election.cc:361] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 1 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c. Message: Invalid argument: T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 1: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:58:46.956357 3569 raft_consensus.cc:1609] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 1: Already voted for candidate 580cbaf7092444dfbc6fb344da88d7b6 in this term. I1010 20:58:46.956415 2921 leader_election.cc:361] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: Vote denied by peer 580cbaf7092444dfbc6fb344da88d7b6. Message: Invalid argument: T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 1: Already voted for candidate 580cbaf7092444dfbc6fb344da88d7b6 in this term. I1010 20:58:46.956565 4007 raft_consensus.cc:1981] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.639s I1010 20:58:46.956584 2747 raft_consensus.cc:1609] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 1: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:58:46.956637 4007 raft_consensus.cc:1823] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given I1010 20:58:46.956650 2920 leader_election.cc:361] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: Vote denied by peer f6d15ccd64ec49be8396ed78e2a88377. Message: Invalid argument: T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 1: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:58:46.956660 2920 leader_election.cc:248] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:46.956673 4006 raft_consensus.cc:1981] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.557s I1010 20:58:46.956678 4002 raft_consensus.cc:1981] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.536s I1010 20:58:46.956683 4006 raft_consensus.cc:1823] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given I1010 20:58:46.956687 4002 raft_consensus.cc:1823] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given I1010 20:58:46.956885 4010 raft_consensus.cc:1981] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.584s I1010 20:58:46.956923 4010 raft_consensus.cc:1823] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given I1010 20:58:46.957146 2629 raft_consensus.cc:1609] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 1: Already voted for candidate 1031512f5120422e9e22a0afabd78f61 in this term. I1010 20:58:46.957237 3037 leader_election.cc:361] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 1 election: Vote denied by peer 1031512f5120422e9e22a0afabd78f61. Message: Invalid argument: T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 1: Already voted for candidate 1031512f5120422e9e22a0afabd78f61 in this term. W1010 20:58:46.986743 3155 leader_election.cc:275] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 42e94fc3adbb419591836d222d946978: Timed out: RequestConsensusVote RPC to 127.6.150.8:35299 timed out after 1.618s I1010 20:58:46.986757 3155 leader_election.cc:248] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:48.369133 3764 raft_consensus.cc:375] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:48.369176 3764 raft_consensus.cc:2031] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:48.434067 3744 raft_consensus.cc:375] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:48.434087 3744 raft_consensus.cc:2031] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:48.440881 3780 raft_consensus.cc:375] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:48.440914 3780 raft_consensus.cc:2031] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Advancing to term 2 W1010 20:58:48.458228 3154 leader_election.cc:275] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 1031512f5120422e9e22a0afabd78f61: Timed out: RequestConsensusVote RPC to 127.6.150.1:46963 timed out after 1.505s W1010 20:58:48.459282 3156 leader_election.cc:275] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 8de6cfa864d64c58bf4eafb689517c93: Timed out: RequestConsensusVote RPC to 127.6.150.3:40117 timed out after 1.505s I1010 20:58:48.459297 3156 leader_election.cc:248] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:48.459480 4012 raft_consensus.cc:1981] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.588s I1010 20:58:48.459498 4012 raft_consensus.cc:1823] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given W1010 20:58:48.502629 2450 leader_election.cc:275] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 7f623b00f5ae4b9c8e8940692d5dfd44: Timed out: RequestConsensusVote RPC to 127.6.150.7:45436 timed out after 1.553s I1010 20:58:48.502647 2450 leader_election.cc:248] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:48.502784 4013 raft_consensus.cc:1981] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.645s I1010 20:58:48.502801 4013 raft_consensus.cc:1823] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given W1010 20:58:48.511705 3388 leader_election.cc:275] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 1031512f5120422e9e22a0afabd78f61: Timed out: RequestConsensusVote RPC to 127.6.150.1:46963 timed out after 1.555s I1010 20:58:48.511734 3388 leader_election.cc:248] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:48.511903 4014 raft_consensus.cc:1981] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.584s I1010 20:58:48.511917 4014 raft_consensus.cc:1823] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given W1010 20:58:48.527921 3038 leader_election.cc:275] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer f6d15ccd64ec49be8396ed78e2a88377: Timed out: RequestConsensusVote RPC to 127.6.150.2:33554 timed out after 1.572s I1010 20:58:48.527942 3038 leader_election.cc:248] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:48.528118 4015 raft_consensus.cc:1981] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.635s I1010 20:58:48.528131 4015 raft_consensus.cc:1823] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given W1010 20:58:48.555863 2568 leader_election.cc:275] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 580cbaf7092444dfbc6fb344da88d7b6: Timed out: RequestConsensusVote RPC to 127.6.150.9:51743 timed out after 1.603s I1010 20:58:48.555878 2568 leader_election.cc:248] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:48.556058 4016 raft_consensus.cc:1981] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.559s I1010 20:58:48.556073 4016 raft_consensus.cc:1823] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given W1010 20:58:48.579445 2921 leader_election.cc:275] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 580cbaf7092444dfbc6fb344da88d7b6: Timed out: RequestConsensusVote RPC to 127.6.150.9:51743 timed out after 1.623s W1010 20:58:48.579463 2921 leader_election.cc:275] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 84bde34331b94c4ea003e48f9317f792: Timed out: RequestConsensusVote RPC to 127.6.150.0:40113 timed out after 1.623s I1010 20:58:48.579478 2921 leader_election.cc:248] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:48.579629 4017 raft_consensus.cc:1981] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.566s I1010 20:58:48.579643 4017 raft_consensus.cc:1823] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given W1010 20:58:48.593622 2801 leader_election.cc:275] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer f6d15ccd64ec49be8396ed78e2a88377: Timed out: RequestConsensusVote RPC to 127.6.150.2:33554 timed out after 1.637s W1010 20:58:48.593653 2801 leader_election.cc:275] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 42e94fc3adbb419591836d222d946978: Timed out: RequestConsensusVote RPC to 127.6.150.8:35299 timed out after 1.637s I1010 20:58:48.593657 2801 leader_election.cc:248] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:48.593848 4018 raft_consensus.cc:1981] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.527s I1010 20:58:48.593863 4018 raft_consensus.cc:1823] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given I1010 20:58:48.598453 3738 raft_consensus.cc:375] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:48.598485 3738 raft_consensus.cc:2031] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:48.618837 3781 raft_consensus.cc:375] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:48.618847 3781 raft_consensus.cc:2031] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:48.630961 3761 raft_consensus.cc:375] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:48.630986 3761 raft_consensus.cc:2031] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:48.631449 3754 raft_consensus.cc:375] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:48.631460 3754 raft_consensus.cc:2031] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:48.632684 3742 raft_consensus.cc:375] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:48.632701 3742 raft_consensus.cc:2031] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Advancing to term 2 W1010 20:58:48.664253 3889 consensus_meta.cc:188] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 1.717s user 0.000s sys 0.000s W1010 20:58:48.664288 3729 consensus_meta.cc:188] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 1.711s user 0.000s sys 0.000s W1010 20:58:48.664320 3692 consensus_meta.cc:188] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 1.717s user 0.000s sys 0.000s W1010 20:58:48.664345 3694 consensus_meta.cc:188] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6: Time spent flushing consensus metadata: real 1.716s user 0.000s sys 0.000s W1010 20:58:48.664402 3869 consensus_meta.cc:188] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.709s user 0.000s sys 0.000s W1010 20:58:48.664436 3867 consensus_meta.cc:188] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 1.712s user 0.000s sys 0.000s W1010 20:58:48.664444 3857 consensus_meta.cc:188] T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 1.709s user 0.000s sys 0.000s W1010 20:58:48.664533 3859 consensus_meta.cc:188] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 1.713s user 0.000s sys 0.000s W1010 20:58:48.664539 3871 consensus_meta.cc:188] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 1.709s user 0.000s sys 0.000s W1010 20:58:48.664600 3755 consensus_meta.cc:188] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 6.323s user 0.000s sys 0.000s I1010 20:58:48.664620 3755 raft_consensus.cc:1981] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.537s W1010 20:58:48.664624 3817 consensus_meta.cc:188] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 1.709s user 0.000s sys 0.000s I1010 20:58:48.664659 3755 raft_consensus.cc:391] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } W1010 20:58:48.664831 3834 consensus_meta.cc:188] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 5.678s user 0.000s sys 0.000s I1010 20:58:48.664845 3834 raft_consensus.cc:1981] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.583s I1010 20:58:48.664865 3834 raft_consensus.cc:391] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } W1010 20:58:48.664876 3815 consensus_meta.cc:188] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 6.049s user 0.000s sys 0.000s I1010 20:58:48.664891 3815 raft_consensus.cc:1981] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.594s I1010 20:58:48.664911 3815 raft_consensus.cc:391] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } W1010 20:58:48.665043 3684 consensus_meta.cc:188] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6: Time spent flushing consensus metadata: real 6.125s user 0.000s sys 0.000s I1010 20:58:48.665082 3684 raft_consensus.cc:1981] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.576s I1010 20:58:48.665105 3684 raft_consensus.cc:391] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } W1010 20:58:48.665132 3842 consensus_meta.cc:188] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 5.353s user 0.000s sys 0.000s I1010 20:58:48.665148 3842 raft_consensus.cc:1981] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.561s I1010 20:58:48.665166 3842 raft_consensus.cc:391] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:58:48.665213 3764 raft_consensus.cc:1981] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.726s I1010 20:58:48.665235 3764 raft_consensus.cc:391] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } W1010 20:58:48.665259 3853 consensus_meta.cc:188] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 6.041s user 0.000s sys 0.000s I1010 20:58:48.665274 3853 raft_consensus.cc:1981] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.550s I1010 20:58:48.665293 3853 raft_consensus.cc:391] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } W1010 20:58:48.665349 3848 consensus_meta.cc:188] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 1.713s user 0.000s sys 0.000s W1010 20:58:48.665496 3881 consensus_meta.cc:188] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 3.300s user 0.000s sys 0.000s W1010 20:58:48.665519 3753 consensus_meta.cc:188] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 3.299s user 0.000s sys 0.000s W1010 20:58:48.665722 3779 consensus_meta.cc:188] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 3.301s user 0.000s sys 0.000s I1010 20:58:48.665879 3781 raft_consensus.cc:1981] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.626s I1010 20:58:48.665915 3781 raft_consensus.cc:391] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } W1010 20:58:48.665927 3863 consensus_meta.cc:188] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 5.805s user 0.000s sys 0.000s I1010 20:58:48.665936 3742 raft_consensus.cc:1981] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.519s I1010 20:58:48.665943 3863 raft_consensus.cc:1981] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.569s I1010 20:58:48.665958 3742 raft_consensus.cc:391] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:58:48.665961 3863 raft_consensus.cc:391] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } I1010 20:58:48.666021 3761 raft_consensus.cc:1981] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.650s I1010 20:58:48.666055 3761 raft_consensus.cc:391] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } I1010 20:58:48.666285 3889 leader_election.cc:223] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:58:48.666337 3889 leader_election.cc:223] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:58:48.666405 3694 leader_election.cc:223] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 1 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:58:48.666446 3694 leader_election.cc:223] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 1 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:58:48.666669 3871 leader_election.cc:223] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 1 election: Requesting vote from peer 580cbaf7092444dfbc6fb344da88d7b6 I1010 20:58:48.666694 3859 leader_election.cc:223] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 1 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:58:48.666703 3871 leader_election.cc:223] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 1 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:58:48.666712 3859 leader_election.cc:223] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 1 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:58:48.666745 3220 raft_consensus.cc:1609] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 1: Already voted for candidate 1380a27d6b824575a3a5e139bee139f4 in this term. I1010 20:58:48.666766 3219 raft_consensus.cc:1654] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 for term 1 because replica is already servicing an update from a current leader or another vote. W1010 20:58:48.666805 3220 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.2:40008 (request call id 1) took 4995ms (client timeout 1564). W1010 20:58:48.666821 3219 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.8:57074 (request call id 1) took 3296ms (client timeout 1512). W1010 20:58:48.666831 3220 rpcz_store.cc:238] Trace: 1010 20:58:43.670955 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:43.670967 (+ 12us) service_pool.cc:202] Handling call 1010 20:58:48.666784 (+4995817us) spinlock_profiling.cc:255] Waited 5 s on lock 0x3b75eb0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f8ce19fe181 00007f8ce08e647c ffffffffffffffff 1010 20:58:48.666788 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":11490190208} W1010 20:58:48.666846 3219 rpcz_store.cc:238] Trace: 1010 20:58:45.370064 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:45.370071 (+ 7us) service_pool.cc:202] Handling call 1010 20:58:48.666783 (+3296712us) spinlock_profiling.cc:255] Waited 3.3 s on lock 0x3b75eb0. stack: 1010 20:58:48.666815 (+ 32us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7582371584,"tcmalloc_contention_cycles":768} I1010 20:58:48.666930 3869 leader_election.cc:223] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 I1010 20:58:48.666952 3982 raft_consensus.cc:1981] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.608s I1010 20:58:48.667021 3982 raft_consensus.cc:1823] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Leader election lost for term 1. Reason: None given I1010 20:58:48.667023 3857 leader_election.cc:223] T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 1 election: Requesting vote from peer 580cbaf7092444dfbc6fb344da88d7b6 I1010 20:58:48.666956 3869 leader_election.cc:223] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:58:48.667037 3857 leader_election.cc:223] T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 1 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 I1010 20:58:48.667146 3100 raft_consensus.cc:1609] T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 in current term 1: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. W1010 20:58:48.667181 3100 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.9:54865 (request call id 1) took 4993ms (client timeout 1648). W1010 20:58:48.667198 3100 rpcz_store.cc:238] Trace: 1010 20:58:43.673816 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:43.673827 (+ 11us) service_pool.cc:202] Handling call 1010 20:58:48.667174 (+4993347us) spinlock_profiling.cc:255] Waited 4.99 s on lock 0x5124470. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007fc8aa54b181 00007fc8a943347c ffffffffffffffff 1010 20:58:48.667177 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":11484592896} I1010 20:58:48.667238 3867 leader_election.cc:223] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 1 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:58:48.667253 3867 leader_election.cc:223] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 1 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:58:48.667410 3729 leader_election.cc:223] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 1 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:58:48.667424 3729 leader_election.cc:223] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 1 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:58:48.667425 2632 raft_consensus.cc:1609] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 1: Already voted for candidate 1031512f5120422e9e22a0afabd78f61 in this term. W1010 20:58:48.667461 2632 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.4:41994 (request call id 1) took 4995ms (client timeout 1526). W1010 20:58:48.667491 2632 rpcz_store.cc:238] Trace: 1010 20:58:43.672316 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:43.672327 (+ 11us) service_pool.cc:202] Handling call 1010 20:58:48.667455 (+4995128us) spinlock_profiling.cc:255] Waited 5 s on lock 0x4340470. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f324f2ae181 00007f324e19647c ffffffffffffffff 1010 20:58:48.667458 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":11488680192} I1010 20:58:48.667507 2631 raft_consensus.cc:1654] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 for term 1 because replica is already servicing an update from a current leader or another vote. W1010 20:58:48.667526 2631 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.8:50170 (request call id 1) took 1711ms (client timeout 1555). W1010 20:58:48.667534 2631 rpcz_store.cc:238] Trace: 1010 20:58:46.956275 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:46.956282 (+ 7us) service_pool.cc:202] Handling call 1010 20:58:48.667521 (+1711239us) spinlock_profiling.cc:255] Waited 1.71 s on lock 0x4340470. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f324f2ae181 00007f324e19647c ffffffffffffffff 1010 20:58:48.667523 (+ 2us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3935798656} I1010 20:58:48.667640 2748 raft_consensus.cc:1609] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 1: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:58:48.667661 3692 leader_election.cc:223] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 1 election: Requesting vote from peer 580cbaf7092444dfbc6fb344da88d7b6 I1010 20:58:48.667667 2513 raft_consensus.cc:1609] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 1: Already voted for candidate 84bde34331b94c4ea003e48f9317f792 in this term. W1010 20:58:48.667673 2748 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.3:58643 (request call id 1) took 1711ms (client timeout 1637). I1010 20:58:48.667677 3692 leader_election.cc:223] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 1 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c W1010 20:58:48.667695 2748 rpcz_store.cc:238] Trace: 1010 20:58:46.956241 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:46.956248 (+ 7us) service_pool.cc:202] Handling call 1010 20:58:48.667666 (+1711418us) spinlock_profiling.cc:255] Waited 1.71 s on lock 0x3419970. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f7357684181 00007f735656c47c ffffffffffffffff 1010 20:58:48.667669 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3936176000} I1010 20:58:48.667703 3848 leader_election.cc:223] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 2 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 W1010 20:58:48.667712 2513 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.4:40901 (request call id 2) took 1711ms (client timeout 1623). I1010 20:58:48.667714 3848 leader_election.cc:223] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 2 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 W1010 20:58:48.667727 2513 rpcz_store.cc:238] Trace: 1010 20:58:46.955764 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:46.955771 (+ 7us) service_pool.cc:202] Handling call 1010 20:58:48.667705 (+1711934us) spinlock_profiling.cc:255] Waited 1.71 s on lock 0x409b970. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007ff4d62cf181 00007ff4d51b747c ffffffffffffffff 1010 20:58:48.667708 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3937330048} I1010 20:58:48.667749 4011 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.646s I1010 20:58:48.667783 4011 raft_consensus.cc:1823] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Leader election lost for term 1. Reason: None given I1010 20:58:48.668047 3779 leader_election.cc:223] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 1 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 I1010 20:58:48.668076 3779 leader_election.cc:223] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 1 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:58:48.668145 3881 leader_election.cc:223] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 1 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:58:48.668154 3881 leader_election.cc:223] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 1 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:58:48.668167 3219 raft_consensus.cc:1584] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 for earlier term 1. Current term is 2. I1010 20:58:48.668182 2986 raft_consensus.cc:1609] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 in current term 1: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:58:48.668198 3453 raft_consensus.cc:1609] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 1: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. W1010 20:58:48.668229 3453 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.3:41009 (request call id 1) took 1711ms (client timeout 1637). W1010 20:58:48.668241 3453 rpcz_store.cc:238] Trace: 1010 20:58:46.956805 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:46.956818 (+ 13us) service_pool.cc:202] Handling call 1010 20:58:48.668223 (+1711405us) spinlock_profiling.cc:255] Waited 1.71 s on lock 0x420c1d0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007fd760afa181 00007fd75f9e247c ffffffffffffffff 1010 20:58:48.668226 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3936123776} I1010 20:58:48.668247 2450 leader_election.cc:361] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 1 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c. Message: Invalid argument: T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 in current term 1: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:58:48.668267 3451 raft_consensus.cc:1654] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 for term 1 because replica is already servicing an update from a current leader or another vote. W1010 20:58:48.668274 3387 leader_election.cc:333] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 1 election: Vote denied by peer 1380a27d6b824575a3a5e139bee139f4 with higher term. Message: Invalid argument: T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 for earlier term 1. Current term is 2. I1010 20:58:48.668283 3387 leader_election.cc:336] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 1 election: Cancelling election due to peer responding with higher term I1010 20:58:48.668346 3452 raft_consensus.cc:1654] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 for term 1 because replica is already servicing an update from a current leader or another vote. I1010 20:58:48.668380 3454 raft_consensus.cc:1609] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 1: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. W1010 20:58:48.668401 3454 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.2:40678 (request call id 2) took 3299ms (client timeout 1538). W1010 20:58:48.668412 3454 rpcz_store.cc:238] Trace: 1010 20:58:45.368915 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:45.368928 (+ 13us) service_pool.cc:202] Handling call 1010 20:58:48.668396 (+3299468us) spinlock_profiling.cc:255] Waited 3.3 s on lock 0x3e4f970. stack: 1010 20:58:48.668399 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7588719488} W1010 20:58:48.668553 3452 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.6:42979 (request call id 1) took 3298ms (client timeout 1618). W1010 20:58:48.668575 3452 rpcz_store.cc:238] Trace: 1010 20:58:45.370102 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:45.370109 (+ 7us) service_pool.cc:202] Handling call 1010 20:58:48.668538 (+3298429us) spinlock_profiling.cc:255] Waited 3.3 s on lock 0x3e4f970. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007fd760afa181 00007fd75f9e247c ffffffffffffffff 1010 20:58:48.668540 (+ 2us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7585404160} I1010 20:58:48.668725 2684 leader_election.cc:361] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 1 election: Vote denied by peer 42e94fc3adbb419591836d222d946978. Message: Service unavailable: T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 for term 1 because replica is already servicing an update from a current leader or another vote. I1010 20:58:48.668836 3452 raft_consensus.cc:1609] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 in current term 1: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:58:48.668874 3454 raft_consensus.cc:2031] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:48.668959 3155 leader_election.cc:361] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 1 election: Vote denied by peer 42e94fc3adbb419591836d222d946978. Message: Invalid argument: T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 in current term 1: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:58:48.669390 3452 raft_consensus.cc:1609] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 1: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:58:48.669432 2567 leader_election.cc:361] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 1 election: Vote denied by peer 42e94fc3adbb419591836d222d946978. Message: Invalid argument: T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 1: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:58:48.669556 3817 leader_election.cc:223] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 I1010 20:58:48.669572 3817 leader_election.cc:223] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:58:48.669608 2744 raft_consensus.cc:1609] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 1: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:58:48.669678 3389 leader_election.cc:361] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 1 election: Vote denied by peer f6d15ccd64ec49be8396ed78e2a88377. Message: Invalid argument: T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 1: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:58:48.669909 3753 leader_election.cc:223] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 I1010 20:58:48.669922 3753 leader_election.cc:223] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 I1010 20:58:48.670089 3569 raft_consensus.cc:1609] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 1: Already voted for candidate 580cbaf7092444dfbc6fb344da88d7b6 in this term. I1010 20:58:48.670121 2866 raft_consensus.cc:1609] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 1: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. I1010 20:58:48.670212 3039 leader_election.cc:361] T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 1 election: Vote denied by peer 580cbaf7092444dfbc6fb344da88d7b6. Message: Invalid argument: T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 1: Already voted for candidate 580cbaf7092444dfbc6fb344da88d7b6 in this term. W1010 20:58:48.670253 2866 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.7:50821 (request call id 1) took 4997ms (client timeout 1590). W1010 20:58:48.670274 2866 rpcz_store.cc:238] Trace: 1010 20:58:43.672777 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:43.672790 (+ 13us) service_pool.cc:202] Handling call 1010 20:58:48.670245 (+4997455us) spinlock_profiling.cc:255] Waited 5 s on lock 0x3f9e470. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f9e2c5b9181 00007f9e2b4a147c ffffffffffffffff 1010 20:58:48.670249 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":11492837760} I1010 20:58:48.670325 2865 raft_consensus.cc:1654] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c for term 1 because replica is already servicing an update from a current leader or another vote. W1010 20:58:48.670358 2865 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.4:46161 (request call id 1) took 3299ms (client timeout 1571). W1010 20:58:48.670373 2865 rpcz_store.cc:238] Trace: 1010 20:58:45.370893 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:45.370926 (+ 33us) service_pool.cc:202] Handling call 1010 20:58:48.670354 (+3299428us) spinlock_profiling.cc:255] Waited 3.3 s on lock 0x3f9e470. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f9e2c5b9181 00007f9e2b4a147c ffffffffffffffff 1010 20:58:48.670356 (+ 2us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7588519168} I1010 20:58:48.670383 2863 raft_consensus.cc:1654] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c for term 2 because replica is already servicing an update from a current leader or another vote. I1010 20:58:48.670521 3569 raft_consensus.cc:1609] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 1: Already voted for candidate 580cbaf7092444dfbc6fb344da88d7b6 in this term. I1010 20:58:48.670567 3390 leader_election.cc:361] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 1 election: Vote denied by peer 580cbaf7092444dfbc6fb344da88d7b6. Message: Invalid argument: T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 1: Already voted for candidate 580cbaf7092444dfbc6fb344da88d7b6 in this term. I1010 20:58:48.670609 3452 raft_consensus.cc:1609] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 in current term 1: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:58:48.670660 3506 leader_election.cc:361] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 1 election: Vote denied by peer 42e94fc3adbb419591836d222d946978. Message: Invalid argument: T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 in current term 1: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:58:48.671038 2863 raft_consensus.cc:1609] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 1: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. I1010 20:58:48.671108 2685 leader_election.cc:361] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 1 election: Vote denied by peer 8de6cfa864d64c58bf4eafb689517c93. Message: Invalid argument: T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 1: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. I1010 20:58:48.671124 2685 leader_election.cc:248] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:48.671162 2921 leader_election.cc:361] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: Vote denied by peer 8de6cfa864d64c58bf4eafb689517c93. Message: Service unavailable: T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c for term 2 because replica is already servicing an update from a current leader or another vote. I1010 20:58:48.671195 2985 raft_consensus.cc:1584] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 for earlier term 1. Current term is 2. W1010 20:58:48.671270 2802 leader_election.cc:333] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c with higher term. Message: Invalid argument: T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 for earlier term 1. Current term is 2. I1010 20:58:48.671283 2802 leader_election.cc:336] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: Cancelling election due to peer responding with higher term I1010 20:58:48.671469 4044 raft_consensus.cc:1981] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.569s I1010 20:58:48.671480 4044 raft_consensus.cc:1823] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given I1010 20:58:48.671658 2865 raft_consensus.cc:1609] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 1: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. I1010 20:58:48.671705 3390 leader_election.cc:361] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 1 election: Vote denied by peer 8de6cfa864d64c58bf4eafb689517c93. Message: Invalid argument: T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 1: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. I1010 20:58:48.671715 3390 leader_election.cc:248] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:48.671713 4045 raft_consensus.cc:1981] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.633s I1010 20:58:48.671720 4045 raft_consensus.cc:1823] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c with higher term. Message: Invalid argument: T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 for earlier term 1. Current term is 2. I1010 20:58:48.671790 4046 raft_consensus.cc:1981] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.594s I1010 20:58:48.671803 4046 raft_consensus.cc:1823] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given I1010 20:58:48.749912 3717 raft_consensus.cc:375] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:48.749943 3717 raft_consensus.cc:2031] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:48.756592 3882 raft_consensus.cc:375] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:48.756604 3882 raft_consensus.cc:2031] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:48.785876 3737 raft_consensus.cc:375] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:48.785887 3737 raft_consensus.cc:2031] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:48.848613 3748 raft_consensus.cc:375] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:48.848629 3748 raft_consensus.cc:2031] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:49.299214 3678 raft_consensus.cc:375] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:49.299235 3678 raft_consensus.cc:2031] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:49.382535 3753 raft_consensus.cc:375] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:49.382552 3753 raft_consensus.cc:2031] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:50.000418 3829 raft_consensus.cc:375] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:50.000437 3829 raft_consensus.cc:2031] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:50.045511 3891 raft_consensus.cc:375] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:50.045526 3891 raft_consensus.cc:2031] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:50.075918 3805 raft_consensus.cc:375] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:50.075951 3805 raft_consensus.cc:2031] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:50.097108 3692 raft_consensus.cc:375] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:50.097149 3692 raft_consensus.cc:2031] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:50.124716 3694 raft_consensus.cc:375] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:50.124730 3694 raft_consensus.cc:2031] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:50.127952 3871 raft_consensus.cc:375] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:50.127964 3871 raft_consensus.cc:2031] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Advancing to term 2 W1010 20:58:50.181066 2802 leader_election.cc:275] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 7f623b00f5ae4b9c8e8940692d5dfd44: Timed out: RequestConsensusVote RPC to 127.6.150.7:45436 timed out after 1.511s W1010 20:58:50.194723 3155 leader_election.cc:275] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer f6d15ccd64ec49be8396ed78e2a88377: Timed out: RequestConsensusVote RPC to 127.6.150.2:33554 timed out after 1.526s I1010 20:58:50.194739 3155 leader_election.cc:248] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:50.195025 4047 raft_consensus.cc:1981] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.632s I1010 20:58:50.195056 4047 raft_consensus.cc:1823] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given W1010 20:58:50.202230 2450 leader_election.cc:275] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 580cbaf7092444dfbc6fb344da88d7b6: Timed out: RequestConsensusVote RPC to 127.6.150.9:51743 timed out after 1.534s I1010 20:58:50.202246 2450 leader_election.cc:248] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:50.204049 3870 raft_consensus.cc:375] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:50.204063 3870 raft_consensus.cc:2031] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Advancing to term 2 W1010 20:58:50.211414 3270 leader_election.cc:275] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 33ef4317e50042ba95e517992caa6531: Timed out: RequestConsensusVote RPC to 127.6.150.5:40764 timed out after 1.545s W1010 20:58:50.213824 3272 leader_election.cc:275] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer f6d15ccd64ec49be8396ed78e2a88377: Timed out: RequestConsensusVote RPC to 127.6.150.2:33554 timed out after 1.545s I1010 20:58:50.213855 3272 leader_election.cc:248] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:50.214072 4049 raft_consensus.cc:1981] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.518s I1010 20:58:50.214088 4049 raft_consensus.cc:1823] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given W1010 20:58:50.253499 3507 leader_election.cc:275] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 8de6cfa864d64c58bf4eafb689517c93: Timed out: RequestConsensusVote RPC to 127.6.150.3:40117 timed out after 1.585s I1010 20:58:50.253512 3507 leader_election.cc:248] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. W1010 20:58:50.265899 3039 leader_election.cc:275] T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 7f623b00f5ae4b9c8e8940692d5dfd44: Timed out: RequestConsensusVote RPC to 127.6.150.7:45436 timed out after 1.598s I1010 20:58:50.265916 3039 leader_election.cc:248] T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:50.266098 4051 raft_consensus.cc:1981] T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.623s I1010 20:58:50.266114 4051 raft_consensus.cc:1823] T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given W1010 20:58:50.271306 2800 leader_election.cc:275] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 1031512f5120422e9e22a0afabd78f61: Timed out: RequestConsensusVote RPC to 127.6.150.1:46963 timed out after 1.600s W1010 20:58:50.271375 2802 leader_election.cc:275] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 84bde34331b94c4ea003e48f9317f792: Timed out: RequestConsensusVote RPC to 127.6.150.0:40113 timed out after 1.600s I1010 20:58:50.271389 2802 leader_election.cc:248] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. W1010 20:58:50.273380 2568 leader_election.cc:275] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer b8062d66fd284035ae3bc2567bc11e7c: Timed out: RequestConsensusVote RPC to 127.6.150.4:44055 timed out after 1.604s I1010 20:58:50.273394 2568 leader_election.cc:248] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:50.273610 4053 raft_consensus.cc:1981] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.638s I1010 20:58:50.273644 4053 raft_consensus.cc:1823] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given W1010 20:58:50.274685 3390 leader_election.cc:275] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 8de6cfa864d64c58bf4eafb689517c93: Timed out: RequestConsensusVote RPC to 127.6.150.3:40117 timed out after 1.607s I1010 20:58:50.274698 3390 leader_election.cc:248] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. W1010 20:58:50.308568 3389 leader_election.cc:275] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer f6d15ccd64ec49be8396ed78e2a88377: Timed out: RequestConsensusVote RPC to 127.6.150.2:33554 timed out after 1.639s I1010 20:58:50.330318 3789 raft_consensus.cc:375] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:50.330353 3789 raft_consensus.cc:2031] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Advancing to term 2 W1010 20:58:50.344944 2921 leader_election.cc:275] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7f623b00f5ae4b9c8e8940692d5dfd44: Timed out: RequestConsensusVote RPC to 127.6.150.7:45436 timed out after 1.675s I1010 20:58:50.344959 2921 leader_election.cc:248] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:50.345176 4055 raft_consensus.cc:1981] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.793s I1010 20:58:50.345207 4055 raft_consensus.cc:1823] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given I1010 20:58:50.349685 3715 raft_consensus.cc:375] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:50.349696 3715 raft_consensus.cc:2031] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:50.358201 3720 raft_consensus.cc:375] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:50.358213 3720 raft_consensus.cc:2031] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Advancing to term 2 W1010 20:58:50.383644 3155 leader_election.cc:275] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 42e94fc3adbb419591836d222d946978: Timed out: RequestConsensusVote RPC to 127.6.150.8:35299 timed out after 1.714s W1010 20:58:50.383674 3155 leader_election.cc:275] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f6d15ccd64ec49be8396ed78e2a88377: Timed out: RequestConsensusVote RPC to 127.6.150.2:33554 timed out after 1.714s I1010 20:58:50.383677 3155 leader_election.cc:248] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:50.383812 4056 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.623s I1010 20:58:50.383836 4056 raft_consensus.cc:1823] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given W1010 20:58:50.393302 3834 consensus_meta.cc:188] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 1.728s user 0.000s sys 0.000s W1010 20:58:50.393316 3755 consensus_meta.cc:188] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 1.729s user 0.000s sys 0.000s W1010 20:58:50.393340 3684 consensus_meta.cc:188] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6: Time spent flushing consensus metadata: real 1.728s user 0.000s sys 0.000s I1010 20:58:50.393460 3834 leader_election.cc:223] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 1 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 W1010 20:58:50.393462 3818 consensus_meta.cc:188] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 3.446s user 0.000s sys 0.000s I1010 20:58:50.393487 3834 leader_election.cc:223] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 1 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 I1010 20:58:50.393501 3684 leader_election.cc:223] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 1 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 I1010 20:58:50.393519 3684 leader_election.cc:223] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 1 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:58:50.393553 3818 leader_election.cc:223] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: Requesting vote from peer 580cbaf7092444dfbc6fb344da88d7b6 I1010 20:58:50.393566 3818 leader_election.cc:223] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 W1010 20:58:50.393726 3763 consensus_meta.cc:188] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 3.677s user 0.000s sys 0.000s I1010 20:58:50.393743 3763 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.563s I1010 20:58:50.393743 3334 raft_consensus.cc:1654] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 for term 1 because replica is already servicing an update from a current leader or another vote. I1010 20:58:50.393764 3763 raft_consensus.cc:391] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } W1010 20:58:50.393795 3334 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.5:60329 (request call id 1) took 1722ms (client timeout 1598). W1010 20:58:50.393815 3334 rpcz_store.cc:238] Trace: 1010 20:58:48.671400 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:48.671423 (+ 23us) service_pool.cc:202] Handling call 1010 20:58:50.393785 (+1722362us) spinlock_profiling.cc:255] Waited 1.72 s on lock 0x3e5bc10. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f3fb2fbf181 00007f3fb1ea747c ffffffffffffffff 1010 20:58:50.393790 (+ 5us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3961309568} W1010 20:58:50.393833 3673 consensus_meta.cc:188] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6: Time spent flushing consensus metadata: real 3.441s user 0.000s sys 0.000s I1010 20:58:50.393857 3673 raft_consensus.cc:1981] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.515s I1010 20:58:50.393887 3673 raft_consensus.cc:391] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } I1010 20:58:50.393971 2744 raft_consensus.cc:1609] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 1: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. W1010 20:58:50.394029 2744 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.7:52494 (request call id 1) took 1722ms (client timeout 1545). W1010 20:58:50.394050 2744 rpcz_store.cc:238] Trace: 1010 20:58:48.671368 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:48.671379 (+ 11us) service_pool.cc:202] Handling call 1010 20:58:50.394014 (+1722635us) spinlock_profiling.cc:255] Waited 1.72 s on lock 0x37cc470. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f7357684181 00007f735656c47c ffffffffffffffff 1010 20:58:50.394025 (+ 11us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3961930624} I1010 20:58:50.394137 3569 raft_consensus.cc:1654] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 for term 1 because replica is already servicing an update from a current leader or another vote. I1010 20:58:50.394186 3829 raft_consensus.cc:1981] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.749s I1010 20:58:50.394224 3829 raft_consensus.cc:391] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } W1010 20:58:50.394261 3569 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.0:33087 (request call id 1) took 1723ms (client timeout 1534). W1010 20:58:50.394289 3569 rpcz_store.cc:238] Trace: 1010 20:58:48.670763 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:48.670793 (+ 30us) service_pool.cc:202] Handling call 1010 20:58:50.394250 (+1723457us) spinlock_profiling.cc:255] Waited 1.72 s on lock 0x44ff6d0. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f6cf36a4181 00007f6cf258c47c ffffffffffffffff 1010 20:58:50.394256 (+ 6us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3963660032} I1010 20:58:50.394341 3755 leader_election.cc:223] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 I1010 20:58:50.394359 3755 leader_election.cc:223] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 I1010 20:58:50.394412 3333 raft_consensus.cc:1654] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 for term 1 because replica is already servicing an update from a current leader or another vote. W1010 20:58:50.394459 3842 consensus_meta.cc:188] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 1.729s user 0.000s sys 0.000s W1010 20:58:50.394462 3333 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.0:38817 (request call id 1) took 3444ms (client timeout 1553). I1010 20:58:50.394489 3570 raft_consensus.cc:1609] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 1: Already voted for candidate 580cbaf7092444dfbc6fb344da88d7b6 in this term. W1010 20:58:50.394491 3333 rpcz_store.cc:238] Trace: 1010 20:58:46.949948 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:46.949996 (+ 48us) service_pool.cc:202] Handling call 1010 20:58:50.394454 (+3444458us) spinlock_profiling.cc:255] Waited 3.44 s on lock 0x3e5b970. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f3fb2fbf181 00007f3fb1ea747c ffffffffffffffff 1010 20:58:50.394458 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7922105600} W1010 20:58:50.394536 3570 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.4:54236 (request call id 1) took 3438ms (client timeout 1623). W1010 20:58:50.394551 3570 rpcz_store.cc:238] Trace: 1010 20:58:46.956314 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:46.956331 (+ 17us) service_pool.cc:202] Handling call 1010 20:58:50.394531 (+3438200us) spinlock_profiling.cc:255] Waited 3.44 s on lock 0x44ff6d0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f6cf36a4181 00007f6cf258c47c ffffffffffffffff 1010 20:58:50.394533 (+ 2us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7907696384} I1010 20:58:50.394601 3842 leader_election.cc:223] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 1 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:58:50.394613 3842 leader_election.cc:223] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 1 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 W1010 20:58:50.394659 3764 consensus_meta.cc:188] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.729s user 0.000s sys 0.000s I1010 20:58:50.394717 3335 raft_consensus.cc:1609] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 1: Already voted for candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in this term. W1010 20:58:50.394745 3815 consensus_meta.cc:188] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 1.730s user 0.000s sys 0.000s I1010 20:58:50.394760 3764 leader_election.cc:223] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 I1010 20:58:50.394773 3764 leader_election.cc:223] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 W1010 20:58:50.394778 3335 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.1:46701 (request call id 1) took 5022ms (client timeout 1559). W1010 20:58:50.394793 3335 rpcz_store.cc:238] Trace: 1010 20:58:45.372197 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:45.372203 (+ 6us) service_pool.cc:202] Handling call 1010 20:58:50.394770 (+5022567us) spinlock_profiling.cc:255] Waited 5.02 s on lock 0x3e5b970. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f3fb2fbf181 00007f3fb1ea747c ffffffffffffffff 1010 20:58:50.394773 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":11551759232} W1010 20:58:50.394809 3781 consensus_meta.cc:188] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 1.729s user 0.000s sys 0.000s I1010 20:58:50.394832 3815 leader_election.cc:223] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 1 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:58:50.394848 3815 leader_election.cc:223] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 1 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:58:50.394876 3781 leader_election.cc:223] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 2 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:58:50.394887 3781 leader_election.cc:223] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 2 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 W1010 20:58:50.394925 3761 consensus_meta.cc:188] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 1.729s user 0.000s sys 0.000s I1010 20:58:50.395004 3761 leader_election.cc:223] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 2 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 I1010 20:58:50.395016 3761 leader_election.cc:223] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 2 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 W1010 20:58:50.395077 3672 consensus_meta.cc:188] T 4aa163baddb74c4998157964dd55b1f4 P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 7.703s user 0.000s sys 0.000s I1010 20:58:50.395092 3672 raft_consensus.cc:1981] T 4aa163baddb74c4998157964dd55b1f4 P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.576s I1010 20:58:50.395110 3672 raft_consensus.cc:391] T 4aa163baddb74c4998157964dd55b1f4 P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } W1010 20:58:50.395206 3808 consensus_meta.cc:188] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 7.044s user 0.000s sys 0.000s I1010 20:58:50.395223 3808 raft_consensus.cc:1981] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.553s I1010 20:58:50.395242 3808 raft_consensus.cc:391] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:58:50.395264 2514 raft_consensus.cc:1609] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 1: Already voted for candidate 84bde34331b94c4ea003e48f9317f792 in this term. W1010 20:58:50.395331 2514 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.3:38215 (request call id 1) took 6722ms (client timeout 1620). W1010 20:58:50.395349 2514 rpcz_store.cc:238] Trace: 1010 20:58:43.672421 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:43.672439 (+ 18us) service_pool.cc:202] Handling call 1010 20:58:50.395323 (+6722884us) spinlock_profiling.cc:255] Waited 6.72 s on lock 0x444c470. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007ff4d62cf181 00007ff4d51b747c ffffffffffffffff 1010 20:58:50.395327 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":15462451968} W1010 20:58:50.395612 3836 consensus_meta.cc:188] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 7.307s user 0.000s sys 0.000s I1010 20:58:50.395629 3836 raft_consensus.cc:1981] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.535s W1010 20:58:50.395630 3882 consensus_meta.cc:188] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.639s user 0.000s sys 0.000s I1010 20:58:50.395644 3882 raft_consensus.cc:1981] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.746s I1010 20:58:50.395649 3836 raft_consensus.cc:391] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:58:50.395663 3882 raft_consensus.cc:391] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } I1010 20:58:50.395819 3805 raft_consensus.cc:1981] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.715s I1010 20:58:50.395838 3805 raft_consensus.cc:391] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:58:50.395864 3720 raft_consensus.cc:1981] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.735s I1010 20:58:50.395885 3720 raft_consensus.cc:391] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:58:50.396077 3789 raft_consensus.cc:1981] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.539s I1010 20:58:50.396100 3789 raft_consensus.cc:391] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:58:50.396477 3099 raft_consensus.cc:1609] T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 1: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. I1010 20:58:50.396565 2512 raft_consensus.cc:2031] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:50.396584 2865 raft_consensus.cc:1584] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 for earlier term 1. Current term is 2. I1010 20:58:50.396627 3270 leader_election.cc:361] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: Vote denied by peer 33ef4317e50042ba95e517992caa6531. Message: Invalid argument: T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 1: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. W1010 20:58:50.396692 3853 consensus_meta.cc:188] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 1.731s user 0.000s sys 0.000s I1010 20:58:50.396697 3452 raft_consensus.cc:2031] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:50.396751 3335 raft_consensus.cc:1609] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 1: Already voted for candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in this term. I1010 20:58:50.396766 3853 leader_election.cc:223] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 1 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:58:50.396777 2630 raft_consensus.cc:1609] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 in current term 1: Already voted for candidate 1031512f5120422e9e22a0afabd78f61 in this term. I1010 20:58:50.396782 3853 leader_election.cc:223] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 1 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 I1010 20:58:50.396806 2685 leader_election.cc:361] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 1 election: Vote denied by peer 7f623b00f5ae4b9c8e8940692d5dfd44. Message: Invalid argument: T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 1: Already voted for candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in this term. W1010 20:58:50.396831 2630 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.6:47001 (request call id 1) took 3442ms (client timeout 1505). W1010 20:58:50.396847 2630 rpcz_store.cc:238] Trace: 1010 20:58:46.954587 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:46.954602 (+ 15us) service_pool.cc:202] Handling call 1010 20:58:50.396820 (+3442218us) spinlock_profiling.cc:255] Waited 3.44 s on lock 0x43401d0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f324f2ae181 00007f324e19647c ffffffffffffffff 1010 20:58:50.396825 (+ 5us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7916959616} W1010 20:58:50.396988 3804 consensus_meta.cc:188] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 7.684s user 0.000s sys 0.000s I1010 20:58:50.397007 3804 raft_consensus.cc:1981] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.561s I1010 20:58:50.397028 3804 raft_consensus.cc:391] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } W1010 20:58:50.397208 2450 leader_election.cc:333] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 1 election: Vote denied by peer 8de6cfa864d64c58bf4eafb689517c93 with higher term. Message: Invalid argument: T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 for earlier term 1. Current term is 2. I1010 20:58:50.397214 2450 leader_election.cc:336] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 1 election: Cancelling election due to peer responding with higher term W1010 20:58:50.397408 3742 consensus_meta.cc:188] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 1.731s user 0.000s sys 0.000s I1010 20:58:50.397492 3742 leader_election.cc:223] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 2 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:58:50.397508 3742 leader_election.cc:223] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 2 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 I1010 20:58:50.397567 2744 raft_consensus.cc:1609] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 1: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:58:50.397608 3099 raft_consensus.cc:1654] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 for term 1 because replica is already servicing an update from a current leader or another vote. W1010 20:58:50.397644 3454 consensus_meta.cc:188] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 1.729s user 0.000s sys 0.000s I1010 20:58:50.397687 3454 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.721s I1010 20:58:50.397696 2682 leader_election.cc:361] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 1 election: Vote denied by peer 33ef4317e50042ba95e517992caa6531. Message: Service unavailable: T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 for term 1 because replica is already servicing an update from a current leader or another vote. I1010 20:58:50.397702 2682 leader_election.cc:248] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:50.397706 3891 raft_consensus.cc:1981] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.605s I1010 20:58:50.397743 3891 raft_consensus.cc:391] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } I1010 20:58:50.397764 3337 raft_consensus.cc:1609] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 in current term 1: Already voted for candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in this term. W1010 20:58:50.397794 3337 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.9:42466 (request call id 1) took 6725ms (client timeout 1648). W1010 20:58:50.397814 3337 rpcz_store.cc:238] Trace: 1010 20:58:43.672338 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:43.672358 (+ 20us) service_pool.cc:202] Handling call 1010 20:58:50.397788 (+6725430us) spinlock_profiling.cc:255] Waited 6.73 s on lock 0x3e5bc10. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f3fb2fbf181 00007f3fb1ea747c ffffffffffffffff 1010 20:58:50.397791 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":15468383104} I1010 20:58:50.398103 3871 raft_consensus.cc:1981] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.673s I1010 20:58:50.398185 3871 raft_consensus.cc:391] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:58:50.398237 2984 raft_consensus.cc:1609] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 in current term 1: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:58:50.398270 3452 raft_consensus.cc:1981] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.704s I1010 20:58:50.398280 3219 raft_consensus.cc:1609] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 1: Already voted for candidate 1380a27d6b824575a3a5e139bee139f4 in this term. I1010 20:58:50.398295 2630 raft_consensus.cc:2031] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:50.398332 2565 leader_election.cc:361] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 1 election: Vote denied by peer 1380a27d6b824575a3a5e139bee139f4. Message: Invalid argument: T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 1: Already voted for candidate 1380a27d6b824575a3a5e139bee139f4 in this term. I1010 20:58:50.398380 3507 leader_election.cc:361] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 1 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c. Message: Invalid argument: T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 in current term 1: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:58:50.398401 2986 raft_consensus.cc:1584] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 for earlier term 1. Current term is 2. W1010 20:58:50.398445 2986 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.1:54805 (request call id 1) took 1728ms (client timeout 1604). W1010 20:58:50.398460 2986 rpcz_store.cc:238] Trace: 1010 20:58:48.669603 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:48.669608 (+ 5us) service_pool.cc:202] Handling call 1010 20:58:50.398438 (+1728830us) spinlock_profiling.cc:255] Waited 1.73 s on lock 0x4a3bc10. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007fe1aafaf181 00007fe1a9e9747c ffffffffffffffff 1010 20:58:50.398442 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3976196864} I1010 20:58:50.398531 3100 raft_consensus.cc:1609] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 1: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. W1010 20:58:50.398578 3100 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.7:37171 (request call id 1) took 1729ms (client timeout 1545). W1010 20:58:50.398598 3100 rpcz_store.cc:238] Trace: 1010 20:58:48.669139 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:48.669144 (+ 5us) service_pool.cc:202] Handling call 1010 20:58:50.398572 (+1729428us) spinlock_profiling.cc:255] Waited 1.73 s on lock 0x51241d0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007fc8aa54b181 00007fc8a943347c ffffffffffffffff 1010 20:58:50.398575 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3977565568} I1010 20:58:50.398799 3337 raft_consensus.cc:1609] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 1: Already voted for candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in this term. I1010 20:58:50.398816 4069 raft_consensus.cc:1981] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.524s I1010 20:58:50.398824 4069 raft_consensus.cc:1823] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given I1010 20:58:50.398859 3038 leader_election.cc:361] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 1 election: Vote denied by peer f6d15ccd64ec49be8396ed78e2a88377. Message: Invalid argument: T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 1: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:58:50.398957 3039 leader_election.cc:361] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 1 election: Vote denied by peer 7f623b00f5ae4b9c8e8940692d5dfd44. Message: Invalid argument: T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 1: Already voted for candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in this term. I1010 20:58:50.398972 3039 leader_election.cc:248] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:50.399116 4070 raft_consensus.cc:1981] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.582s I1010 20:58:50.399142 4070 raft_consensus.cc:1823] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given I1010 20:58:50.399480 2510 raft_consensus.cc:1609] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 1: Already voted for candidate 84bde34331b94c4ea003e48f9317f792 in this term. I1010 20:58:50.399565 3273 leader_election.cc:361] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: Vote denied by peer 84bde34331b94c4ea003e48f9317f792. Message: Invalid argument: T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 1: Already voted for candidate 84bde34331b94c4ea003e48f9317f792 in this term. I1010 20:58:50.409389 3698 raft_consensus.cc:375] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:50.409433 3698 raft_consensus.cc:2031] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:50.423848 3722 raft_consensus.cc:375] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:50.423859 3722 raft_consensus.cc:2031] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:50.441853 3894 raft_consensus.cc:375] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:50.441865 3894 raft_consensus.cc:2031] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:50.617169 3697 raft_consensus.cc:375] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:50.617182 3697 raft_consensus.cc:2031] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:51.059681 3818 raft_consensus.cc:375] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:51.059703 3818 raft_consensus.cc:2031] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:51.741966 3742 raft_consensus.cc:375] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:51.741984 3742 raft_consensus.cc:2031] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Advancing to term 3 I1010 20:58:51.790662 3671 raft_consensus.cc:375] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:51.790711 3671 raft_consensus.cc:2031] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:51.830994 3853 raft_consensus.cc:375] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:51.831010 3853 raft_consensus.cc:2031] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:51.868331 3761 raft_consensus.cc:375] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:51.868346 3761 raft_consensus.cc:2031] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Advancing to term 3 I1010 20:58:51.885387 3837 raft_consensus.cc:375] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:51.885399 3837 raft_consensus.cc:2031] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:51.911747 3755 raft_consensus.cc:375] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:51.911775 3755 raft_consensus.cc:2031] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:51.912241 3729 raft_consensus.cc:375] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:51.912257 3729 raft_consensus.cc:2031] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Advancing to term 2 W1010 20:58:51.917513 3390 leader_election.cc:275] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b8062d66fd284035ae3bc2567bc11e7c: Timed out: RequestConsensusVote RPC to 127.6.150.4:44055 timed out after 1.519s W1010 20:58:51.917563 3390 leader_election.cc:275] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7f623b00f5ae4b9c8e8940692d5dfd44: Timed out: RequestConsensusVote RPC to 127.6.150.7:45436 timed out after 1.519s I1010 20:58:51.917572 3390 leader_election.cc:248] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. W1010 20:58:51.923307 3763 consensus_meta.cc:188] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 1.530s user 0.000s sys 0.000s W1010 20:58:51.923369 3673 consensus_meta.cc:188] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6: Time spent flushing consensus metadata: real 1.529s user 0.000s sys 0.000s W1010 20:58:51.923470 3819 consensus_meta.cc:188] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 4.970s user 0.000s sys 0.000s I1010 20:58:51.923588 3763 leader_election.cc:223] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 2 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 I1010 20:58:51.923599 3819 leader_election.cc:223] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 1 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:58:51.923609 3763 leader_election.cc:223] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 2 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:58:51.923611 3819 leader_election.cc:223] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 1 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 I1010 20:58:51.923631 2748 raft_consensus.cc:1609] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 in current term 2: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. W1010 20:58:51.923712 2748 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.6:49570 (request call id 3) took 3255ms (client timeout 1714). W1010 20:58:51.923728 2748 rpcz_store.cc:238] Trace: 1010 20:58:48.667852 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:48.667860 (+ 8us) service_pool.cc:202] Handling call 1010 20:58:51.923693 (+3255833us) spinlock_profiling.cc:255] Waited 3.26 s on lock 0x3419eb0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f7357684181 00007f735656c47c ffffffffffffffff 1010 20:58:51.923699 (+ 6us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7488245504} I1010 20:58:51.923753 3219 raft_consensus.cc:1609] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 2: Already voted for candidate 1380a27d6b824575a3a5e139bee139f4 in this term. I1010 20:58:51.923804 3979 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.801s I1010 20:58:51.923840 3979 raft_consensus.cc:1823] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Leader election lost for term 1. Reason: None given I1010 20:58:51.923859 2745 raft_consensus.cc:1654] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 for term 1 because replica is already servicing an update from a current leader or another vote. W1010 20:58:51.923867 3737 consensus_meta.cc:188] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6: Time spent flushing consensus metadata: real 3.138s user 0.000s sys 0.000s I1010 20:58:51.923897 3737 raft_consensus.cc:1981] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.681s W1010 20:58:51.923899 2745 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.8:42100 (request call id 2) took 3254ms (client timeout 1639). W1010 20:58:51.923907 3738 consensus_meta.cc:188] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 3.325s user 0.000s sys 0.000s I1010 20:58:51.923921 3738 raft_consensus.cc:1981] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.780s W1010 20:58:51.923918 3829 consensus_meta.cc:188] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 1.530s user 0.000s sys 0.000s W1010 20:58:51.923926 2745 rpcz_store.cc:238] Trace: 1010 20:58:48.669548 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:48.669576 (+ 28us) service_pool.cc:202] Handling call 1010 20:58:51.923893 (+3254317us) spinlock_profiling.cc:255] Waited 3.25 s on lock 0x3419eb0. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f7357684181 00007f735656c47c ffffffffffffffff 1010 20:58:51.923896 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7484818688} I1010 20:58:51.923928 3737 raft_consensus.cc:391] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:58:51.923977 3738 raft_consensus.cc:391] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } I1010 20:58:51.924026 3829 leader_election.cc:223] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 2 election: Requesting vote from peer 580cbaf7092444dfbc6fb344da88d7b6 I1010 20:58:51.924057 3829 leader_election.cc:223] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 2 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 I1010 20:58:51.924239 2749 raft_consensus.cc:1609] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 1: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:58:51.924273 2747 raft_consensus.cc:1654] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 for term 1 because replica is already servicing an update from a current leader or another vote. W1010 20:58:51.924283 2749 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.1:40603 (request call id 1) took 8251ms (client timeout 1576). W1010 20:58:51.924315 2747 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.5:40398 (request call id 1) took 4967ms (client timeout 1572). W1010 20:58:51.924320 2749 rpcz_store.cc:238] Trace: 1010 20:58:43.672343 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:43.672361 (+ 18us) service_pool.cc:202] Handling call 1010 20:58:51.924274 (+8251913us) spinlock_profiling.cc:255] Waited 8.25 s on lock 0x37cc1d0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f7357684181 00007f735656c47c ffffffffffffffff 1010 20:58:51.924278 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":18979243904} I1010 20:58:51.924322 3673 leader_election.cc:223] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 2 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 W1010 20:58:51.924331 2747 rpcz_store.cc:238] Trace: 1010 20:58:46.957035 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:46.957045 (+ 10us) service_pool.cc:202] Handling call 1010 20:58:51.924309 (+4967264us) spinlock_profiling.cc:255] Waited 4.97 s on lock 0x37cc1d0. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f7357684181 00007f735656c47c ffffffffffffffff 1010 20:58:51.924312 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":11424599424} I1010 20:58:51.924347 3673 leader_election.cc:223] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 2 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 W1010 20:58:51.924374 3870 consensus_meta.cc:188] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 1.720s user 0.000s sys 0.000s I1010 20:58:51.924418 3870 raft_consensus.cc:1981] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.503s I1010 20:58:51.924477 3870 raft_consensus.cc:391] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } W1010 20:58:51.924721 3871 consensus_meta.cc:188] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 1.527s user 0.000s sys 0.000s I1010 20:58:51.924751 2629 raft_consensus.cc:2031] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:51.924868 3871 leader_election.cc:223] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 2 election: Requesting vote from peer 580cbaf7092444dfbc6fb344da88d7b6 I1010 20:58:51.924913 3871 leader_election.cc:223] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 2 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:58:51.924912 3999 raft_consensus.cc:1981] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.544s I1010 20:58:51.924891 4054 raft_consensus.cc:1981] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.719s I1010 20:58:51.924937 3571 raft_consensus.cc:1584] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 for earlier term 1. Current term is 2. I1010 20:58:51.924952 4054 raft_consensus.cc:1823] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Leader election lost for term 1. Reason: None given W1010 20:58:51.924983 3571 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.1:54035 (request call id 1) took 4971ms (client timeout 1603). I1010 20:58:51.924993 3337 raft_consensus.cc:1609] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 in current term 2: Already voted for candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in this term. W1010 20:58:51.925015 3571 rpcz_store.cc:238] Trace: 1010 20:58:46.953905 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:46.953916 (+ 11us) service_pool.cc:202] Handling call 1010 20:58:51.924973 (+4971057us) spinlock_profiling.cc:255] Waited 4.97 s on lock 0x44ff430. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f6cf36a4181 00007f6cf258c47c ffffffffffffffff 1010 20:58:51.924978 (+ 5us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":11433308928} I1010 20:58:51.925047 3569 raft_consensus.cc:1609] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 2: Already voted for candidate 580cbaf7092444dfbc6fb344da88d7b6 in this term. I1010 20:58:51.925079 3999 raft_consensus.cc:1823] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Leader election lost for term 1. Reason: None given I1010 20:58:51.925094 2682 leader_election.cc:361] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 2 election: Vote denied by peer 1380a27d6b824575a3a5e139bee139f4. Message: Invalid argument: T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 2: Already voted for candidate 1380a27d6b824575a3a5e139bee139f4 in this term. I1010 20:58:51.925122 3273 leader_election.cc:361] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 2 election: Vote denied by peer 580cbaf7092444dfbc6fb344da88d7b6. Message: Invalid argument: T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 2: Already voted for candidate 580cbaf7092444dfbc6fb344da88d7b6 in this term. I1010 20:58:51.925182 3507 leader_election.cc:361] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 2 election: Vote denied by peer 7f623b00f5ae4b9c8e8940692d5dfd44. Message: Invalid argument: T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 in current term 2: Already voted for candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in this term. I1010 20:58:51.925189 3099 raft_consensus.cc:1609] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 1: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. W1010 20:58:51.925333 3672 consensus_meta.cc:188] T 4aa163baddb74c4998157964dd55b1f4 P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 1.530s user 0.000s sys 0.000s I1010 20:58:51.925418 3672 leader_election.cc:223] T 4aa163baddb74c4998157964dd55b1f4 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 1 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:58:51.925448 2513 raft_consensus.cc:1609] T 4aa163baddb74c4998157964dd55b1f4 P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 1: Already voted for candidate 84bde34331b94c4ea003e48f9317f792 in this term. I1010 20:58:51.925456 3672 leader_election.cc:223] T 4aa163baddb74c4998157964dd55b1f4 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 1 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 W1010 20:58:51.925508 2513 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.3:38215 (request call id 2) took 3254ms (client timeout 1600). W1010 20:58:51.925524 2513 rpcz_store.cc:238] Trace: 1010 20:58:48.671006 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:48.671013 (+ 7us) service_pool.cc:202] Handling call 1010 20:58:51.925481 (+3254468us) spinlock_profiling.cc:255] Waited 3.25 s on lock 0x409b6d0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007ff4d62cf181 00007ff4d51b747c ffffffffffffffff 1010 20:58:51.925484 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7485141248} W1010 20:58:51.925720 3754 consensus_meta.cc:188] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 3.294s user 0.000s sys 0.000s I1010 20:58:51.925753 3754 raft_consensus.cc:1981] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.521s I1010 20:58:51.925786 3754 raft_consensus.cc:391] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } W1010 20:58:51.926015 3808 consensus_meta.cc:188] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 1.531s user 0.000s sys 0.000s I1010 20:58:51.926115 2682 leader_election.cc:361] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 1 election: Vote denied by peer 33ef4317e50042ba95e517992caa6531. Message: Invalid argument: T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 1: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. I1010 20:58:51.926139 3808 leader_election.cc:223] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 1 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 I1010 20:58:51.926165 3808 leader_election.cc:223] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 1 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:58:51.926180 3100 raft_consensus.cc:1654] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 for term 1 because replica is already servicing an update from a current leader or another vote. W1010 20:58:51.926239 3100 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.0:46745 (request call id 3) took 1527ms (client timeout 1594). W1010 20:58:51.926272 3100 rpcz_store.cc:238] Trace: 1010 20:58:50.398662 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:50.398749 (+ 87us) service_pool.cc:202] Handling call 1010 20:58:51.926226 (+1527477us) spinlock_profiling.cc:255] Waited 1.53 s on lock 0x4d75eb0. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007fc8aa54b181 00007fc8a943347c ffffffffffffffff 1010 20:58:51.926235 (+ 9us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3513065216} I1010 20:58:51.926312 3101 raft_consensus.cc:1654] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 for term 2 because replica is already servicing an update from a current leader or another vote. W1010 20:58:51.926323 3780 consensus_meta.cc:188] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 3.485s user 0.000s sys 0.000s W1010 20:58:51.926332 3101 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.3:53121 (request call id 2) took 1530ms (client timeout 1626). I1010 20:58:51.926352 3780 raft_consensus.cc:1981] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.703s W1010 20:58:51.926357 3101 rpcz_store.cc:238] Trace: 1010 20:58:50.396296 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:50.396320 (+ 24us) service_pool.cc:202] Handling call 1010 20:58:51.926328 (+1530008us) spinlock_profiling.cc:255] Waited 1.53 s on lock 0x4d75eb0. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007fc8aa54b181 00007fc8a943347c ffffffffffffffff 1010 20:58:51.926330 (+ 2us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3518962944} I1010 20:58:51.926386 3780 raft_consensus.cc:391] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } I1010 20:58:51.926401 3103 raft_consensus.cc:1609] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 1: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. W1010 20:58:51.926425 3103 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.3:53121 (request call id 1) took 8253ms (client timeout 1620). W1010 20:58:51.926455 3103 rpcz_store.cc:238] Trace: 1010 20:58:43.673177 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:43.673189 (+ 12us) service_pool.cc:202] Handling call 1010 20:58:51.926421 (+8253232us) spinlock_profiling.cc:255] Waited 8.25 s on lock 0x4d75eb0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007fc8aa54b181 00007fc8a943347c ffffffffffffffff 1010 20:58:51.926423 (+ 2us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":18982289664} I1010 20:58:51.926544 2799 leader_election.cc:361] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 2 election: Vote denied by peer 33ef4317e50042ba95e517992caa6531. Message: Service unavailable: T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 for term 2 because replica is already servicing an update from a current leader or another vote. I1010 20:58:51.926901 2447 leader_election.cc:361] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 1 election: Vote denied by peer 33ef4317e50042ba95e517992caa6531. Message: Service unavailable: T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 for term 1 because replica is already servicing an update from a current leader or another vote. I1010 20:58:51.927456 2860 raft_consensus.cc:1584] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 for earlier term 1. Current term is 2. W1010 20:58:51.927573 3039 leader_election.cc:333] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 1 election: Vote denied by peer 8de6cfa864d64c58bf4eafb689517c93 with higher term. Message: Invalid argument: T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 for earlier term 1. Current term is 2. I1010 20:58:51.927584 3039 leader_election.cc:336] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 1 election: Cancelling election due to peer responding with higher term I1010 20:58:51.927717 4076 raft_consensus.cc:1981] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.634s I1010 20:58:51.927743 4076 raft_consensus.cc:1823] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: Vote denied by peer 8de6cfa864d64c58bf4eafb689517c93 with higher term. Message: Invalid argument: T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 for earlier term 1. Current term is 2. W1010 20:58:51.928689 3836 consensus_meta.cc:188] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 1.533s user 0.000s sys 0.000s W1010 20:58:51.928710 3882 consensus_meta.cc:188] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.533s user 0.000s sys 0.000s I1010 20:58:51.928858 3836 leader_election.cc:223] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: Requesting vote from peer 580cbaf7092444dfbc6fb344da88d7b6 I1010 20:58:51.928870 3836 leader_election.cc:223] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:58:51.928889 3882 leader_election.cc:223] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:58:51.928896 2866 raft_consensus.cc:1609] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 in current term 1: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. I1010 20:58:51.928910 3882 leader_election.cc:223] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 W1010 20:58:51.928913 3863 consensus_meta.cc:188] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 3.263s user 0.000s sys 0.000s W1010 20:58:51.928939 2866 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.9:41775 (request call id 1) took 3258ms (client timeout 1585). W1010 20:58:51.928952 2866 rpcz_store.cc:238] Trace: 1010 20:58:48.670364 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:48.670377 (+ 13us) service_pool.cc:202] Handling call 1010 20:58:51.928932 (+3258555us) spinlock_profiling.cc:255] Waited 3.26 s on lock 0x3f9e710. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f9e2c5b9181 00007f9e2b4a147c ffffffffffffffff 1010 20:58:51.928935 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7494569344} I1010 20:58:51.929016 2863 raft_consensus.cc:1654] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 for term 1 because replica is already servicing an update from a current leader or another vote. I1010 20:58:51.929026 3863 leader_election.cc:223] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 I1010 20:58:51.929054 3863 leader_election.cc:223] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 W1010 20:58:51.929071 2863 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.8:60818 (request call id 1) took 3257ms (client timeout 1607). I1010 20:58:51.929093 2865 raft_consensus.cc:1654] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 for term 1 because replica is already servicing an update from a current leader or another vote. W1010 20:58:51.929095 2863 rpcz_store.cc:238] Trace: 1010 20:58:48.671619 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:48.671639 (+ 20us) service_pool.cc:202] Handling call 1010 20:58:51.929050 (+3257411us) spinlock_profiling.cc:255] Waited 3.26 s on lock 0x3f9e710. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f9e2c5b9181 00007f9e2b4a147c ffffffffffffffff 1010 20:58:51.929059 (+ 9us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7491946240} W1010 20:58:51.929121 3753 consensus_meta.cc:188] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 2.547s user 0.000s sys 0.000s I1010 20:58:51.929131 2861 raft_consensus.cc:1654] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 for term 2 because replica is already servicing an update from a current leader or another vote. I1010 20:58:51.929137 3753 raft_consensus.cc:1981] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.574s W1010 20:58:51.929152 2865 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.1:58515 (request call id 1) took 1531ms (client timeout 1550). W1010 20:58:51.929177 2865 rpcz_store.cc:238] Trace: 1010 20:58:50.397299 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:50.397311 (+ 12us) service_pool.cc:202] Handling call 1010 20:58:51.929137 (+1531826us) spinlock_profiling.cc:255] Waited 1.53 s on lock 0x3f9e9b0. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f9e2c5b9181 00007f9e2b4a147c ffffffffffffffff 1010 20:58:51.929140 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3523069312,"tcmalloc_contention_cycles":1792} I1010 20:58:51.929181 3753 raft_consensus.cc:391] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } I1010 20:58:51.929311 2864 raft_consensus.cc:1609] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 in current term 1: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. W1010 20:58:51.929339 2864 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.6:47110 (request call id 1) took 4974ms (client timeout 1505). W1010 20:58:51.929358 2864 rpcz_store.cc:238] Trace: 1010 20:58:46.954472 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:46.954515 (+ 43us) service_pool.cc:202] Handling call 1010 20:58:51.929333 (+4974818us) spinlock_profiling.cc:255] Waited 4.97 s on lock 0x3f9e9b0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f9e2c5b9181 00007f9e2b4a147c ffffffffffffffff 1010 20:58:51.929336 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":11441949952} I1010 20:58:51.929512 2568 leader_election.cc:361] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 1 election: Vote denied by peer 8de6cfa864d64c58bf4eafb689517c93. Message: Service unavailable: T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 for term 1 because replica is already servicing an update from a current leader or another vote. I1010 20:58:51.929522 2568 leader_election.cc:248] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:51.929605 3453 raft_consensus.cc:1584] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 for earlier term 1. Current term is 2. I1010 20:58:51.929658 3390 leader_election.cc:361] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 2 election: Vote denied by peer 8de6cfa864d64c58bf4eafb689517c93. Message: Service unavailable: T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 for term 2 because replica is already servicing an update from a current leader or another vote. W1010 20:58:51.929723 2801 leader_election.cc:333] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: Vote denied by peer 42e94fc3adbb419591836d222d946978 with higher term. Message: Invalid argument: T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 for earlier term 1. Current term is 2. I1010 20:58:51.929730 2801 leader_election.cc:336] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: Cancelling election due to peer responding with higher term I1010 20:58:51.929944 4079 raft_consensus.cc:1981] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.559s I1010 20:58:51.929996 4079 raft_consensus.cc:1823] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: Vote denied by peer 42e94fc3adbb419591836d222d946978 with higher term. Message: Invalid argument: T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 for earlier term 1. Current term is 2. W1010 20:58:51.930044 3720 consensus_meta.cc:188] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.534s user 0.000s sys 0.000s I1010 20:58:51.930166 3720 leader_election.cc:223] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:58:51.930178 2985 raft_consensus.cc:1609] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 2: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:58:51.930198 3720 leader_election.cc:223] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 W1010 20:58:51.930229 2985 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.8:46146 (request call id 3) took 1532ms (client timeout 1519). W1010 20:58:51.930244 2985 rpcz_store.cc:238] Trace: 1010 20:58:50.397591 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:50.397600 (+ 9us) service_pool.cc:202] Handling call 1010 20:58:51.930216 (+1532616us) spinlock_profiling.cc:255] Waited 1.53 s on lock 0x4a3b970. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007fe1aafaf181 00007fe1a9e9747c ffffffffffffffff 1010 20:58:51.930225 (+ 9us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3524892032} W1010 20:58:51.930297 3805 consensus_meta.cc:188] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.534s user 0.000s sys 0.000s I1010 20:58:51.930384 3805 leader_election.cc:223] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 I1010 20:58:51.930402 3805 leader_election.cc:223] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 W1010 20:58:51.930507 3894 consensus_meta.cc:188] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.489s user 0.000s sys 0.000s I1010 20:58:51.930531 3894 raft_consensus.cc:1981] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.690s I1010 20:58:51.930583 3894 raft_consensus.cc:391] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } W1010 20:58:51.930753 3789 consensus_meta.cc:188] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 1.535s user 0.000s sys 0.000s I1010 20:58:51.930898 3789 leader_election.cc:223] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 2 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:58:51.930919 3789 leader_election.cc:223] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 2 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:58:51.930925 3220 raft_consensus.cc:1609] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 2: Already voted for candidate 1380a27d6b824575a3a5e139bee139f4 in this term. I1010 20:58:51.931013 2985 raft_consensus.cc:1609] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 in current term 2: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:58:51.931077 3156 leader_election.cc:361] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 2 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c. Message: Invalid argument: T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 in current term 2: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:58:51.931133 3761 raft_consensus.cc:1981] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.584s I1010 20:58:51.931169 3761 raft_consensus.cc:391] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } I1010 20:58:51.931434 2918 leader_election.cc:361] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: Vote denied by peer 1380a27d6b824575a3a5e139bee139f4. Message: Invalid argument: T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 2: Already voted for candidate 1380a27d6b824575a3a5e139bee139f4 in this term. W1010 20:58:51.931951 3804 consensus_meta.cc:188] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 1.535s user 0.000s sys 0.000s I1010 20:58:51.932085 3804 leader_election.cc:223] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 1 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:58:51.932102 3804 leader_election.cc:223] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 1 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 I1010 20:58:51.932142 2631 raft_consensus.cc:1609] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 1: Already voted for candidate 1031512f5120422e9e22a0afabd78f61 in this term. W1010 20:58:51.932183 2631 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.3:51480 (request call id 1) took 3260ms (client timeout 1600). W1010 20:58:51.932212 3748 consensus_meta.cc:188] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 3.084s user 0.000s sys 0.000s W1010 20:58:51.932231 2631 rpcz_store.cc:238] Trace: 1010 20:58:48.671762 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:48.671769 (+ 7us) service_pool.cc:202] Handling call 1010 20:58:51.932176 (+3260407us) spinlock_profiling.cc:255] Waited 3.26 s on lock 0x3f85eb0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f324f2ae181 00007f324e19647c ffffffffffffffff 1010 20:58:51.932179 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7498827520} I1010 20:58:51.932236 3748 raft_consensus.cc:1981] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.762s I1010 20:58:51.932257 3748 raft_consensus.cc:391] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } I1010 20:58:51.932263 2627 raft_consensus.cc:1654] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 for term 1 because replica is already servicing an update from a current leader or another vote. I1010 20:58:51.932265 2509 raft_consensus.cc:1609] T 4aa163baddb74c4998157964dd55b1f4 P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 1: Already voted for candidate 84bde34331b94c4ea003e48f9317f792 in this term. I1010 20:58:51.932335 2568 leader_election.cc:361] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 1 election: Vote denied by peer 84bde34331b94c4ea003e48f9317f792. Message: Invalid argument: T 4aa163baddb74c4998157964dd55b1f4 P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 1: Already voted for candidate 84bde34331b94c4ea003e48f9317f792 in this term. I1010 20:58:51.932377 2448 leader_election.cc:361] T 4aa163baddb74c4998157964dd55b1f4 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 1 election: Vote denied by peer 1031512f5120422e9e22a0afabd78f61. Message: Service unavailable: T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 for term 1 because replica is already servicing an update from a current leader or another vote. W1010 20:58:51.932262 3715 consensus_meta.cc:188] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 1.583s user 0.000s sys 0.000s I1010 20:58:51.932464 3715 raft_consensus.cc:1981] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.720s I1010 20:58:51.932487 3715 raft_consensus.cc:391] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } W1010 20:58:51.933466 3271 leader_election.cc:275] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 1031512f5120422e9e22a0afabd78f61: Timed out: RequestConsensusVote RPC to 127.6.150.1:46963 timed out after 1.537s I1010 20:58:51.933480 3271 leader_election.cc:248] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. W1010 20:58:51.933782 3454 consensus_meta.cc:188] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 1.536s user 0.000s sys 0.000s I1010 20:58:51.933799 3454 raft_consensus.cc:1677] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1380a27d6b824575a3a5e139bee139f4 in term 2. W1010 20:58:51.933815 3454 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.6:42979 (request call id 3) took 3265ms (client timeout 1714). W1010 20:58:51.933851 3454 rpcz_store.cc:238] Trace: 1010 20:58:48.668795 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:48.668870 (+ 75us) service_pool.cc:202] Handling call 1010 20:58:51.933811 (+3264941us) inbound_call.cc:130] Queueing success response Metrics: {"tcmalloc_contention_cycles":896} I1010 20:58:51.933874 3779 raft_consensus.cc:375] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:51.933907 3779 raft_consensus.cc:2031] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Advancing to term 3 W1010 20:58:51.933950 3452 consensus_meta.cc:188] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 1.536s user 0.000s sys 0.000s I1010 20:58:51.933969 3452 raft_consensus.cc:1677] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8062d66fd284035ae3bc2567bc11e7c in term 2. W1010 20:58:51.933977 3452 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.4:56561 (request call id 3) took 1537ms (client timeout 1726). W1010 20:58:51.933990 3452 rpcz_store.cc:238] Trace: 1010 20:58:50.396653 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:50.396682 (+ 29us) service_pool.cc:202] Handling call 1010 20:58:51.933973 (+1537291us) inbound_call.cc:130] Queueing success response Metrics: {"tcmalloc_contention_cycles":1920} I1010 20:58:51.934043 2920 leader_election.cc:347] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: Vote granted by peer 42e94fc3adbb419591836d222d946978 I1010 20:58:51.934051 2920 leader_election.cc:248] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I1010 20:58:51.934170 4081 raft_consensus.cc:1981] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.520s I1010 20:58:51.934180 4081 raft_consensus.cc:1861] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Leader election won for term 2 W1010 20:58:51.933893 3891 consensus_meta.cc:188] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 1.536s user 0.000s sys 0.000s I1010 20:58:51.934304 3891 leader_election.cc:223] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 2 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:58:51.934329 3891 leader_election.cc:223] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 2 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 I1010 20:58:51.934417 3220 raft_consensus.cc:2031] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:51.934444 4081 raft_consensus.cc:467] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 2 LEADER]: Becoming Leader. State: Replica: b8062d66fd284035ae3bc2567bc11e7c, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:51.934484 4081 consensus_queue.cc:151] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 2, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } I1010 20:58:51.936164 2403 catalog_manager.cc:2018] T 84fcfff9fdef4acf8c2b47f36330567f reported consensus state change: term changed from 0 to 2, leader changed from to b8062d66fd284035ae3bc2567bc11e7c (127.6.150.4). New consensus state: current_term: 2 leader_uuid: "b8062d66fd284035ae3bc2567bc11e7c" config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } } W1010 20:58:51.949332 3273 leader_election.cc:275] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 580cbaf7092444dfbc6fb344da88d7b6: Timed out: RequestConsensusVote RPC to 127.6.150.9:51743 timed out after 1.552s I1010 20:58:51.949345 3273 leader_election.cc:248] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. W1010 20:58:51.971318 3507 leader_election.cc:275] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 84bde34331b94c4ea003e48f9317f792: Timed out: RequestConsensusVote RPC to 127.6.150.0:40113 timed out after 1.576s I1010 20:58:51.971329 3507 leader_election.cc:248] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:51.971532 4089 raft_consensus.cc:1981] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.501s I1010 20:58:51.971550 4089 raft_consensus.cc:1823] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given I1010 20:58:51.972985 3898 raft_consensus.cc:375] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:51.972996 3898 raft_consensus.cc:2031] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:51.989109 3817 raft_consensus.cc:375] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:51.989137 3817 raft_consensus.cc:2031] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Advancing to term 2 W1010 20:58:52.022490 2802 leader_election.cc:275] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 84bde34331b94c4ea003e48f9317f792: Timed out: RequestConsensusVote RPC to 127.6.150.0:40113 timed out after 1.626s I1010 20:58:52.022512 2802 leader_election.cc:248] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:52.022687 4090 raft_consensus.cc:1981] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.566s I1010 20:58:52.022701 4090 raft_consensus.cc:1823] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given W1010 20:58:52.045752 3153 leader_election.cc:275] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 33ef4317e50042ba95e517992caa6531: Timed out: RequestConsensusVote RPC to 127.6.150.5:40764 timed out after 1.650s W1010 20:58:52.046715 3156 leader_election.cc:275] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 84bde34331b94c4ea003e48f9317f792: Timed out: RequestConsensusVote RPC to 127.6.150.0:40113 timed out after 1.650s I1010 20:58:52.046728 3156 leader_election.cc:248] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:52.073577 3794 raft_consensus.cc:375] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:52.073604 3794 raft_consensus.cc:2031] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:52.104848 3713 raft_consensus.cc:375] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:52.104861 3713 raft_consensus.cc:2031] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Advancing to term 2 W1010 20:58:52.125185 2919 leader_election.cc:275] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1031512f5120422e9e22a0afabd78f61: Timed out: RequestConsensusVote RPC to 127.6.150.1:46963 timed out after 1.726s I1010 20:58:52.220203 3863 raft_consensus.cc:375] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:52.220235 3863 raft_consensus.cc:2031] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:52.269722 3881 raft_consensus.cc:375] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:52.269768 3881 raft_consensus.cc:2031] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:52.434936 3452 raft_consensus.cc:857] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Refusing update from remote peer b8062d66fd284035ae3bc2567bc11e7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch) I1010 20:58:52.435031 4081 consensus_queue.cc:652] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [LEADER]: Connected to new peer: Peer: 42e94fc3adbb419591836d222d946978, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:58:52.730701 3819 raft_consensus.cc:375] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:52.730717 3819 raft_consensus.cc:2031] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Advancing to term 2 W1010 20:58:53.435320 2919 consensus_peers.cc:332] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 84fcfff9fdef4acf8c2b47f36330567f. Status: Timed out: UpdateConsensus RPC to 127.6.150.1:46963 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 1 times. W1010 20:58:53.440454 3505 leader_election.cc:275] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1031512f5120422e9e22a0afabd78f61: Timed out: RequestConsensusVote RPC to 127.6.150.1:46963 timed out after 1.515s I1010 20:58:53.440476 3505 leader_election.cc:248] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:53.440723 4098 raft_consensus.cc:1981] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.732s I1010 20:58:53.440759 4098 raft_consensus.cc:1823] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given W1010 20:58:53.465598 2802 leader_election.cc:275] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 580cbaf7092444dfbc6fb344da88d7b6: Timed out: RequestConsensusVote RPC to 127.6.150.9:51743 timed out after 1.535s W1010 20:58:53.471019 3156 leader_election.cc:275] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8de6cfa864d64c58bf4eafb689517c93: Timed out: RequestConsensusVote RPC to 127.6.150.3:40117 timed out after 1.539s I1010 20:58:53.471035 3156 leader_election.cc:248] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:53.471174 4099 raft_consensus.cc:1981] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.748s I1010 20:58:53.471192 4099 raft_consensus.cc:1823] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given W1010 20:58:53.480849 3039 leader_election.cc:275] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 84bde34331b94c4ea003e48f9317f792: Timed out: RequestConsensusVote RPC to 127.6.150.0:40113 timed out after 1.553s I1010 20:58:53.487098 3834 raft_consensus.cc:375] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:53.487112 3834 raft_consensus.cc:2031] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 1 FOLLOWER]: Advancing to term 2 W1010 20:58:53.487855 2684 leader_election.cc:275] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 42e94fc3adbb419591836d222d946978: Timed out: RequestConsensusVote RPC to 127.6.150.8:35299 timed out after 1.563s I1010 20:58:53.487866 2684 leader_election.cc:248] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:53.488072 4100 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.649s I1010 20:58:53.488092 4100 raft_consensus.cc:1823] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given W1010 20:58:53.494035 2568 leader_election.cc:275] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 8de6cfa864d64c58bf4eafb689517c93: Timed out: RequestConsensusVote RPC to 127.6.150.3:40117 timed out after 1.561s I1010 20:58:53.494052 2568 leader_election.cc:248] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:53.494215 4101 raft_consensus.cc:1981] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.627s I1010 20:58:53.494233 4101 raft_consensus.cc:1823] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given W1010 20:58:53.499368 2799 leader_election.cc:275] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 1380a27d6b824575a3a5e139bee139f4: Timed out: RequestConsensusVote RPC to 127.6.150.6:46774 timed out after 1.569s W1010 20:58:53.499527 2800 leader_election.cc:275] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 1031512f5120422e9e22a0afabd78f61: Timed out: RequestConsensusVote RPC to 127.6.150.1:46963 timed out after 1.569s I1010 20:58:53.499562 2800 leader_election.cc:248] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. W1010 20:58:53.502621 2450 leader_election.cc:275] T 4aa163baddb74c4998157964dd55b1f4 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 8de6cfa864d64c58bf4eafb689517c93: Timed out: RequestConsensusVote RPC to 127.6.150.3:40117 timed out after 1.576s I1010 20:58:53.502637 2450 leader_election.cc:248] T 4aa163baddb74c4998157964dd55b1f4 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:53.502794 4103 raft_consensus.cc:1981] T 4aa163baddb74c4998157964dd55b1f4 P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.562s I1010 20:58:53.502813 4103 raft_consensus.cc:1823] T 4aa163baddb74c4998157964dd55b1f4 P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given W1010 20:58:53.540344 3389 leader_election.cc:275] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f6d15ccd64ec49be8396ed78e2a88377: Timed out: RequestConsensusVote RPC to 127.6.150.2:33554 timed out after 1.605s W1010 20:58:53.540539 3387 leader_election.cc:275] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1380a27d6b824575a3a5e139bee139f4: Timed out: RequestConsensusVote RPC to 127.6.150.6:46774 timed out after 1.605s I1010 20:58:53.540554 3387 leader_election.cc:248] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:53.540693 4104 raft_consensus.cc:1981] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.636s I1010 20:58:53.540709 4104 raft_consensus.cc:1823] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given W1010 20:58:53.548104 2683 leader_election.cc:275] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 1031512f5120422e9e22a0afabd78f61: Timed out: RequestConsensusVote RPC to 127.6.150.1:46963 timed out after 1.623s I1010 20:58:53.548141 2683 leader_election.cc:248] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I1010 20:58:53.573181 3889 raft_consensus.cc:375] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:53.573196 3889 raft_consensus.cc:2031] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Advancing to term 2 W1010 20:58:53.599568 3390 leader_election.cc:275] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 580cbaf7092444dfbc6fb344da88d7b6: Timed out: RequestConsensusVote RPC to 127.6.150.9:51743 timed out after 1.673s I1010 20:58:53.599582 3390 leader_election.cc:248] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:53.599725 4106 raft_consensus.cc:1981] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.689s I1010 20:58:53.599742 4106 raft_consensus.cc:1823] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given W1010 20:58:53.631286 3894 consensus_meta.cc:188] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.701s user 0.000s sys 0.000s W1010 20:58:53.631288 3753 consensus_meta.cc:188] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 1.702s user 0.000s sys 0.000s W1010 20:58:53.631347 3870 consensus_meta.cc:188] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 1.707s user 0.000s sys 0.000s W1010 20:58:53.631352 3737 consensus_meta.cc:188] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6: Time spent flushing consensus metadata: real 1.707s user 0.000s sys 0.000s W1010 20:58:53.631362 3738 consensus_meta.cc:188] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 1.707s user 0.000s sys 0.000s W1010 20:58:53.631577 3722 consensus_meta.cc:188] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6: Time spent flushing consensus metadata: real 3.208s user 0.000s sys 0.000s W1010 20:58:53.631584 3744 consensus_meta.cc:188] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 5.197s user 0.000s sys 0.000s I1010 20:58:53.631605 3722 raft_consensus.cc:1981] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.657s I1010 20:58:53.631611 3744 raft_consensus.cc:1981] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.534s I1010 20:58:53.631644 3722 raft_consensus.cc:391] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } I1010 20:58:53.631646 3744 raft_consensus.cc:391] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } W1010 20:58:53.631647 3898 consensus_meta.cc:188] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.659s user 0.000s sys 0.000s I1010 20:58:53.631671 3898 raft_consensus.cc:1981] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.562s I1010 20:58:53.631700 3898 raft_consensus.cc:391] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } W1010 20:58:53.631702 3697 consensus_meta.cc:188] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 3.015s user 0.000s sys 0.000s I1010 20:58:53.631718 3697 raft_consensus.cc:1981] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.569s I1010 20:58:53.631750 3697 raft_consensus.cc:391] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } W1010 20:58:53.631779 3694 consensus_meta.cc:188] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6: Time spent flushing consensus metadata: real 3.507s user 0.000s sys 0.000s I1010 20:58:53.631809 3694 raft_consensus.cc:1981] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.564s I1010 20:58:53.631855 3694 raft_consensus.cc:391] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } W1010 20:58:53.632104 3779 consensus_meta.cc:188] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 1.698s user 0.000s sys 0.000s I1010 20:58:53.632135 3779 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.757s I1010 20:58:53.632160 3779 raft_consensus.cc:391] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } W1010 20:58:53.632213 3698 consensus_meta.cc:188] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 3.223s user 0.000s sys 0.000s I1010 20:58:53.632261 3698 raft_consensus.cc:1981] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.608s I1010 20:58:53.632287 3698 raft_consensus.cc:391] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } I1010 20:58:53.632355 3737 leader_election.cc:223] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 2 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:58:53.632385 3737 leader_election.cc:223] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 2 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 I1010 20:58:53.632411 3570 raft_consensus.cc:1584] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 for earlier term 1. Current term is 2. W1010 20:58:53.632470 3570 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.7:48285 (request call id 2) took 3235ms (client timeout 1552). W1010 20:58:53.632504 3570 rpcz_store.cc:238] Trace: 1010 20:58:50.397303 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:50.397312 (+ 9us) service_pool.cc:202] Handling call 1010 20:58:53.632461 (+3235149us) spinlock_profiling.cc:255] Waited 3.24 s on lock 0x44ffeb0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f6cf36a4181 00007f6cf258c47c ffffffffffffffff 1010 20:58:53.632466 (+ 5us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7440698752} I1010 20:58:53.632567 3894 leader_election.cc:223] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: Requesting vote from peer 580cbaf7092444dfbc6fb344da88d7b6 I1010 20:58:53.632599 3894 leader_election.cc:223] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:58:53.632696 3101 raft_consensus.cc:2031] T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:53.632758 3870 leader_election.cc:223] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 2 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:58:53.632766 3335 raft_consensus.cc:1609] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 2: Already voted for candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in this term. I1010 20:58:53.632773 3870 leader_election.cc:223] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 2 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c W1010 20:58:53.632779 3761 consensus_meta.cc:188] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 1.702s user 0.000s sys 0.000s W1010 20:58:53.632819 3335 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.8:39867 (request call id 2) took 3235ms (client timeout 1519). W1010 20:58:53.632839 3335 rpcz_store.cc:238] Trace: 1010 20:58:50.397573 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:50.397626 (+ 53us) service_pool.cc:202] Handling call 1010 20:58:53.632811 (+3235185us) spinlock_profiling.cc:255] Waited 3.24 s on lock 0x42161d0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f3fb2fbf181 00007f3fb1ea747c ffffffffffffffff 1010 20:58:53.632815 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7440793856} W1010 20:58:53.632870 3748 consensus_meta.cc:188] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 1.701s user 0.000s sys 0.000s I1010 20:58:53.632889 3738 leader_election.cc:223] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 2 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 I1010 20:58:53.632949 3738 leader_election.cc:223] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 2 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:58:53.632982 3337 raft_consensus.cc:1654] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c for term 2 because replica is already servicing an update from a current leader or another vote. W1010 20:58:53.633003 3337 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.4:46735 (request call id 4) took 1702ms (client timeout 1735). W1010 20:58:53.633014 3337 rpcz_store.cc:238] Trace: 1010 20:58:51.930480 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:51.930504 (+ 24us) service_pool.cc:202] Handling call 1010 20:58:53.632999 (+1702495us) spinlock_profiling.cc:255] Waited 1.7 s on lock 0x42161d0. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f3fb2fbf181 00007f3fb1ea747c ffffffffffffffff 1010 20:58:53.633001 (+ 2us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3915651328} I1010 20:58:53.633147 3753 leader_election.cc:223] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 2 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 I1010 20:58:53.633167 3753 leader_election.cc:223] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 2 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 I1010 20:58:53.633194 2862 raft_consensus.cc:1584] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 for earlier term 1. Current term is 2. I1010 20:58:53.633220 3748 leader_election.cc:223] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 2 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 W1010 20:58:53.633230 2862 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.0:39483 (request call id 2) took 1707ms (client timeout 1576). I1010 20:58:53.633236 3748 leader_election.cc:223] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 2 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 W1010 20:58:53.633245 2862 rpcz_store.cc:238] Trace: 1010 20:58:51.925568 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:51.925581 (+ 13us) service_pool.cc:202] Handling call 1010 20:58:53.633223 (+1707642us) spinlock_profiling.cc:255] Waited 1.71 s on lock 0x3c2deb0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f9e2c5b9181 00007f9e2b4a147c ffffffffffffffff 1010 20:58:53.633226 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3927473920} W1010 20:58:53.633246 3837 consensus_meta.cc:188] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 1.748s user 0.000s sys 0.000s I1010 20:58:53.633261 3837 raft_consensus.cc:1981] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.591s I1010 20:58:53.633289 3837 raft_consensus.cc:391] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } I1010 20:58:53.633303 2921 leader_election.cc:361] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: Vote denied by peer 7f623b00f5ae4b9c8e8940692d5dfd44. Message: Service unavailable: T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c for term 2 because replica is already servicing an update from a current leader or another vote. I1010 20:58:53.633431 2631 raft_consensus.cc:2031] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:53.633443 4052 raft_consensus.cc:1981] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.726s I1010 20:58:53.633466 3761 leader_election.cc:223] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 3 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 I1010 20:58:53.633471 4052 raft_consensus.cc:1823] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Leader election lost for term 1. Reason: None given I1010 20:58:53.633479 3761 leader_election.cc:223] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 3 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:58:53.633483 2865 raft_consensus.cc:1654] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 for term 1 because replica is already servicing an update from a current leader or another vote. W1010 20:58:53.633517 2865 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.1:58515 (request call id 2) took 1701ms (client timeout 1561). W1010 20:58:53.633528 2865 rpcz_store.cc:238] Trace: 1010 20:58:51.932193 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:51.932210 (+ 17us) service_pool.cc:202] Handling call 1010 20:58:53.633505 (+1701295us) spinlock_profiling.cc:255] Waited 1.7 s on lock 0x3c2deb0. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f9e2c5b9181 00007f9e2b4a147c ffffffffffffffff 1010 20:58:53.633508 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3912912640} I1010 20:58:53.633527 2628 raft_consensus.cc:1584] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 for earlier term 1. Current term is 2. W1010 20:58:53.633582 2628 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.2:58461 (request call id 1) took 1707ms (client timeout 1623). W1010 20:58:53.633600 2628 rpcz_store.cc:238] Trace: 1010 20:58:51.925638 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:51.925661 (+ 23us) service_pool.cc:202] Handling call 1010 20:58:53.633574 (+1707913us) spinlock_profiling.cc:255] Waited 1.71 s on lock 0x3f85c10. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f324f2ae181 00007f324e19647c ffffffffffffffff 1010 20:58:53.633577 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3927995648} W1010 20:58:53.633643 3754 consensus_meta.cc:188] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 1.708s user 0.000s sys 0.000s W1010 20:58:53.633653 3715 consensus_meta.cc:188] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 1.701s user 0.000s sys 0.000s I1010 20:58:53.633682 2985 raft_consensus.cc:1609] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 2: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:58:53.633747 3715 leader_election.cc:223] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 2 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 I1010 20:58:53.633752 3454 raft_consensus.cc:1609] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 2: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:58:53.633759 3754 leader_election.cc:223] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 2 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:58:53.633765 3715 leader_election.cc:223] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 2 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 I1010 20:58:53.633774 3754 leader_election.cc:223] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 2 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 I1010 20:58:53.633770 2509 raft_consensus.cc:1654] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 for term 3 because replica is already servicing an update from a current leader or another vote. I1010 20:58:53.633797 3273 leader_election.cc:361] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 2 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c. Message: Invalid argument: T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 2: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:58:53.633865 2509 raft_consensus.cc:2031] T 4aa163baddb74c4998157964dd55b1f4 P 84bde34331b94c4ea003e48f9317f792 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:53.633869 3156 leader_election.cc:361] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 3 election: Vote denied by peer 84bde34331b94c4ea003e48f9317f792. Message: Service unavailable: T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 for term 3 because replica is already servicing an update from a current leader or another vote. I1010 20:58:53.633891 2511 raft_consensus.cc:1609] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 in current term 2: Already voted for candidate 84bde34331b94c4ea003e48f9317f792 in this term. I1010 20:58:53.633894 4091 raft_consensus.cc:1981] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.744s W1010 20:58:53.633898 2629 consensus_meta.cc:188] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 1.709s user 0.000s sys 0.000s I1010 20:58:53.633925 4091 raft_consensus.cc:1823] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: None given I1010 20:58:53.633929 2629 raft_consensus.cc:1981] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.711s W1010 20:58:53.633951 2511 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.6:59570 (request call id 2) took 3236ms (client timeout 1650). W1010 20:58:53.633987 2511 rpcz_store.cc:238] Trace: 1010 20:58:50.397902 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:50.397928 (+ 26us) service_pool.cc:202] Handling call 1010 20:58:53.633940 (+3236012us) spinlock_profiling.cc:255] Waited 3.24 s on lock 0x409beb0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007ff4d62cf181 00007ff4d51b747c ffffffffffffffff 1010 20:58:53.633946 (+ 6us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7442623744} I1010 20:58:53.634130 3217 raft_consensus.cc:1584] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 for earlier term 2. Current term is 3. I1010 20:58:53.634141 2632 raft_consensus.cc:1584] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 for earlier term 1. Current term is 2. I1010 20:58:53.634160 2509 raft_consensus.cc:1981] T 4aa163baddb74c4998157964dd55b1f4 P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.702s W1010 20:58:53.634202 2632 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.7:33193 (request call id 2) took 3239ms (client timeout 1537). W1010 20:58:53.634222 2632 rpcz_store.cc:238] Trace: 1010 20:58:50.395041 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:50.395049 (+ 8us) service_pool.cc:202] Handling call 1010 20:58:53.634182 (+3239133us) spinlock_profiling.cc:255] Waited 3.24 s on lock 0x3f85970. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f324f2ae181 00007f324e19647c ffffffffffffffff 1010 20:58:53.634186 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7449860992} W1010 20:58:53.634220 2447 leader_election.cc:333] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 2 election: Vote denied by peer 1380a27d6b824575a3a5e139bee139f4 with higher term. Message: Invalid argument: T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 for earlier term 2. Current term is 3. I1010 20:58:53.634227 2447 leader_election.cc:336] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term I1010 20:58:53.634300 3779 leader_election.cc:223] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 3 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 I1010 20:58:53.634320 3779 leader_election.cc:223] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 3 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 W1010 20:58:53.634398 3742 consensus_meta.cc:188] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 1.892s user 0.000s sys 0.000s I1010 20:58:53.634416 3742 raft_consensus.cc:1981] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.614s I1010 20:58:53.634434 3742 raft_consensus.cc:391] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:58:53.634479 4107 raft_consensus.cc:1981] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.631s I1010 20:58:53.634487 4107 raft_consensus.cc:1823] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 1380a27d6b824575a3a5e139bee139f4 with higher term. Message: Invalid argument: T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 for earlier term 2. Current term is 3. I1010 20:58:53.634541 3217 raft_consensus.cc:2031] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Advancing to term 3 I1010 20:58:53.634544 2684 leader_election.cc:361] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 2 election: Vote denied by peer 42e94fc3adbb419591836d222d946978. Message: Invalid argument: T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 2: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:58:53.634624 2747 raft_consensus.cc:1654] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 for term 2 because replica is already servicing an update from a current leader or another vote. I1010 20:58:53.634694 3451 raft_consensus.cc:1654] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 for term 2 because replica is already servicing an update from a current leader or another vote. W1010 20:58:53.634704 2747 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.8:42100 (request call id 4) took 1700ms (client timeout 1605). W1010 20:58:53.634718 2747 rpcz_store.cc:238] Trace: 1010 20:58:51.934367 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:51.934374 (+ 7us) service_pool.cc:202] Handling call 1010 20:58:53.634697 (+1700323us) spinlock_profiling.cc:255] Waited 1.7 s on lock 0x3419c10. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f7357684181 00007f735656c47c ffffffffffffffff 1010 20:58:53.634701 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3910549248} W1010 20:58:53.634728 3451 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.2:40678 (request call id 4) took 1710ms (client timeout 1563). I1010 20:58:53.634727 2748 raft_consensus.cc:2031] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Advancing to term 3 I1010 20:58:53.634747 2746 raft_consensus.cc:1584] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 for earlier term 1. Current term is 2. W1010 20:58:53.634757 3451 rpcz_store.cc:238] Trace: 1010 20:58:51.923979 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:51.923987 (+ 8us) service_pool.cc:202] Handling call 1010 20:58:53.634721 (+1710734us) spinlock_profiling.cc:255] Waited 1.71 s on lock 0x3e4f970. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007fd760afa181 00007fd75f9e247c ffffffffffffffff 1010 20:58:53.634724 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3934594304} W1010 20:58:53.634770 2746 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.6:49570 (request call id 2) took 4967ms (client timeout 1526). W1010 20:58:53.634784 2746 rpcz_store.cc:238] Trace: 1010 20:58:48.667050 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:48.667066 (+ 16us) service_pool.cc:202] Handling call 1010 20:58:53.634765 (+4967699us) spinlock_profiling.cc:255] Waited 4.97 s on lock 0x3419c10. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f7357684181 00007f735656c47c ffffffffffffffff 1010 20:58:53.634768 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":11425589504} I1010 20:58:53.634791 4038 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.728s I1010 20:58:53.634806 4038 raft_consensus.cc:1823] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Leader election lost for term 1. Reason: Vote denied by peer 1380a27d6b824575a3a5e139bee139f4 with higher term. Message: Invalid argument: T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 for earlier term 1. Current term is 2. W1010 20:58:53.636016 3780 consensus_meta.cc:188] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 1.710s user 0.000s sys 0.000s I1010 20:58:53.636168 3780 leader_election.cc:223] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 2 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:58:53.636195 3780 leader_election.cc:223] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 2 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 I1010 20:58:53.636195 3103 raft_consensus.cc:1609] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 2: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. W1010 20:58:53.636261 3103 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.4:56288 (request call id 2) took 1707ms (client timeout 1746). W1010 20:58:53.636277 3103 rpcz_store.cc:238] Trace: 1010 20:58:51.929008 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:51.929014 (+ 6us) service_pool.cc:202] Handling call 1010 20:58:53.636246 (+1707232us) spinlock_profiling.cc:255] Waited 1.71 s on lock 0x4d75c10. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007fc8aa54b181 00007fc8a943347c ffffffffffffffff 1010 20:58:53.636250 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3926459136} I1010 20:58:53.636368 2918 leader_election.cc:361] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: Vote denied by peer 33ef4317e50042ba95e517992caa6531. Message: Invalid argument: T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 2: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. W1010 20:58:53.636385 3678 consensus_meta.cc:188] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 4.337s user 0.000s sys 0.000s I1010 20:58:53.636416 2985 raft_consensus.cc:1609] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 2: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. W1010 20:58:53.636450 3713 consensus_meta.cc:188] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 1.532s user 0.000s sys 0.000s I1010 20:58:53.636471 3713 raft_consensus.cc:1981] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.726s I1010 20:58:53.636498 3713 raft_consensus.cc:391] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } I1010 20:58:53.636525 3039 leader_election.cc:361] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 2 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c. Message: Invalid argument: T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 2: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:58:53.636451 3678 raft_consensus.cc:1981] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.668s I1010 20:58:53.636701 3678 raft_consensus.cc:391] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } I1010 20:58:53.636749 3101 raft_consensus.cc:1981] T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.695s W1010 20:58:53.646559 2921 leader_election.cc:275] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8de6cfa864d64c58bf4eafb689517c93: Timed out: RequestConsensusVote RPC to 127.6.150.3:40117 timed out after 1.715s I1010 20:58:53.646574 2921 leader_election.cc:248] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:53.646724 4108 raft_consensus.cc:1981] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.715s I1010 20:58:53.646738 4108 raft_consensus.cc:1823] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given W1010 20:58:53.666914 2920 leader_election.cc:275] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 42e94fc3adbb419591836d222d946978: Timed out: RequestConsensusVote RPC to 127.6.150.8:35299 timed out after 1.735s I1010 20:58:53.666927 2920 leader_election.cc:248] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:53.667057 4109 raft_consensus.cc:1981] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.503s I1010 20:58:53.667075 4109 raft_consensus.cc:1823] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given W1010 20:58:53.676656 3271 leader_election.cc:275] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1031512f5120422e9e22a0afabd78f61: Timed out: RequestConsensusVote RPC to 127.6.150.1:46963 timed out after 1.749s I1010 20:58:53.676671 3271 leader_election.cc:248] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:53.676810 4110 raft_consensus.cc:1981] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.538s I1010 20:58:53.676827 4110 raft_consensus.cc:1823] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given W1010 20:58:53.676905 2921 leader_election.cc:275] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 84bde34331b94c4ea003e48f9317f792: Timed out: RequestConsensusVote RPC to 127.6.150.0:40113 timed out after 1.746s I1010 20:58:53.676913 2921 leader_election.cc:248] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:53.677083 4111 raft_consensus.cc:1981] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.615s I1010 20:58:53.677104 4111 raft_consensus.cc:1823] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given I1010 20:58:53.739284 3842 raft_consensus.cc:375] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:53.739296 3842 raft_consensus.cc:2031] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:53.786857 3869 raft_consensus.cc:375] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:53.786875 3869 raft_consensus.cc:2031] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Advancing to term 3 W1010 20:58:54.936518 2919 consensus_peers.cc:332] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 84fcfff9fdef4acf8c2b47f36330567f. Status: Timed out: UpdateConsensus RPC to 127.6.150.1:46963 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 2 times. I1010 20:58:55.002167 3870 raft_consensus.cc:375] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:55.002187 3870 raft_consensus.cc:2031] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Advancing to term 3 W1010 20:58:55.136628 3272 leader_election.cc:275] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 42e94fc3adbb419591836d222d946978: Timed out: RequestConsensusVote RPC to 127.6.150.8:35299 timed out after 1.503s I1010 20:58:55.136648 3272 leader_election.cc:248] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. W1010 20:58:55.156023 2447 leader_election.cc:275] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 33ef4317e50042ba95e517992caa6531: Timed out: RequestConsensusVote RPC to 127.6.150.5:40764 timed out after 1.521s W1010 20:58:55.159099 3698 consensus_meta.cc:188] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 1.527s user 0.000s sys 0.000s W1010 20:58:55.159101 3694 consensus_meta.cc:188] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6: Time spent flushing consensus metadata: real 1.527s user 0.000s sys 0.000s W1010 20:58:55.159128 3898 consensus_meta.cc:188] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.527s user 0.000s sys 0.000s I1010 20:58:55.159482 3898 leader_election.cc:223] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: Requesting vote from peer 580cbaf7092444dfbc6fb344da88d7b6 I1010 20:58:55.159505 3898 leader_election.cc:223] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 I1010 20:58:55.159543 3698 leader_election.cc:223] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 2 election: Requesting vote from peer 580cbaf7092444dfbc6fb344da88d7b6 I1010 20:58:55.159561 3698 leader_election.cc:223] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 2 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:58:55.159592 3694 leader_election.cc:223] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 2 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:58:55.159611 3694 leader_election.cc:223] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 2 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:58:55.159675 4050 raft_consensus.cc:1981] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.787s I1010 20:58:55.159713 4050 raft_consensus.cc:1823] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Leader election lost for term 1. Reason: None given I1010 20:58:55.159734 3569 raft_consensus.cc:1609] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 2: Already voted for candidate 580cbaf7092444dfbc6fb344da88d7b6 in this term. W1010 20:58:55.159764 3569 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.8:57555 (request call id 2) took 3233ms (client timeout 1673). W1010 20:58:55.159783 3569 rpcz_store.cc:238] Trace: 1010 20:58:51.926340 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:51.926357 (+ 17us) service_pool.cc:202] Handling call 1010 20:58:55.159754 (+3233397us) spinlock_profiling.cc:255] Waited 3.23 s on lock 0x44ff970. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f6cf36a4181 00007f6cf258c47c ffffffffffffffff 1010 20:58:55.159759 (+ 5us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7436741504} I1010 20:58:55.159834 3571 raft_consensus.cc:1654] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 for term 1 because replica is already servicing an update from a current leader or another vote. W1010 20:58:55.159862 3571 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.3:43794 (request call id 1) took 3229ms (client timeout 1535). W1010 20:58:55.159874 3571 rpcz_store.cc:238] Trace: 1010 20:58:51.930064 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:51.930085 (+ 21us) service_pool.cc:202] Handling call 1010 20:58:55.159856 (+3229771us) spinlock_profiling.cc:255] Waited 3.23 s on lock 0x44ff970. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f6cf36a4181 00007f6cf258c47c ffffffffffffffff 1010 20:58:55.159858 (+ 2us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7428399488} I1010 20:58:55.159976 2865 raft_consensus.cc:2031] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:55.160027 2749 raft_consensus.cc:1609] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 2: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. W1010 20:58:55.160063 2749 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.4:44654 (request call id 2) took 1527ms (client timeout 1690). I1010 20:58:55.160063 3451 raft_consensus.cc:1609] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 in current term 2: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. W1010 20:58:55.160080 2749 rpcz_store.cc:238] Trace: 1010 20:58:53.632723 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:53.632732 (+ 9us) service_pool.cc:202] Handling call 1010 20:58:55.160056 (+1527324us) spinlock_profiling.cc:255] Waited 1.53 s on lock 0x34196d0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f7357684181 00007f735656c47c ffffffffffffffff 1010 20:58:55.160060 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3512751488} I1010 20:58:55.160147 2920 leader_election.cc:361] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: Vote denied by peer f6d15ccd64ec49be8396ed78e2a88377. Message: Invalid argument: T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 2: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:58:55.160197 3506 leader_election.cc:361] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 2 election: Vote denied by peer 42e94fc3adbb419591836d222d946978. Message: Invalid argument: T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 in current term 2: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:58:55.160229 2985 raft_consensus.cc:1609] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 2: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:58:55.160280 2685 leader_election.cc:361] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 2 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c. Message: Invalid argument: T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 2: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:58:55.160315 3569 raft_consensus.cc:2031] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Advancing to term 2 W1010 20:58:55.164333 2629 consensus_meta.cc:188] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 1.530s user 0.000s sys 0.000s I1010 20:58:55.164353 2629 raft_consensus.cc:1677] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 580cbaf7092444dfbc6fb344da88d7b6 in term 2. W1010 20:58:55.164355 3869 consensus_meta.cc:188] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.377s user 0.000s sys 0.000s W1010 20:58:55.164368 2629 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.9:55298 (request call id 2) took 3239ms (client timeout 1515). I1010 20:58:55.164376 3869 raft_consensus.cc:1981] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.996s W1010 20:58:55.164389 2629 rpcz_store.cc:238] Trace: 1010 20:58:51.924715 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:51.924731 (+ 16us) service_pool.cc:202] Handling call 1010 20:58:55.164364 (+3239633us) inbound_call.cc:130] Queueing success response Metrics: {"tcmalloc_contention_cycles":3456} I1010 20:58:55.164403 3869 raft_consensus.cc:391] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } W1010 20:58:55.164531 2630 consensus_meta.cc:188] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 4.766s user 0.000s sys 0.000s I1010 20:58:55.164546 2630 raft_consensus.cc:1981] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.616s W1010 20:58:55.164644 3729 consensus_meta.cc:188] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 3.252s user 0.000s sys 0.000s W1010 20:58:55.164645 3697 consensus_meta.cc:188] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 1.533s user 0.000s sys 0.000s I1010 20:58:55.164662 3729 raft_consensus.cc:1981] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.512s I1010 20:58:55.164681 3729 raft_consensus.cc:391] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } I1010 20:58:55.164726 3697 leader_election.cc:223] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 2 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 I1010 20:58:55.164739 3697 leader_election.cc:223] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 2 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:58:55.164741 3667 raft_consensus.cc:375] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:55.164749 3667 raft_consensus.cc:2031] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Advancing to term 3 W1010 20:58:55.164930 3671 consensus_meta.cc:188] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 3.374s user 0.000s sys 0.000s I1010 20:58:55.164952 3671 raft_consensus.cc:1981] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.562s W1010 20:58:55.164957 3834 consensus_meta.cc:188] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 1.678s user 0.000s sys 0.000s I1010 20:58:55.164970 3671 raft_consensus.cc:391] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } I1010 20:58:55.164971 3834 raft_consensus.cc:1981] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.573s I1010 20:58:55.164995 3834 raft_consensus.cc:391] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:58:55.165139 2864 raft_consensus.cc:1609] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 2: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. W1010 20:58:55.165206 2864 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.4:46161 (request call id 4) took 3234ms (client timeout 1715). W1010 20:58:55.165211 3722 consensus_meta.cc:188] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6: Time spent flushing consensus metadata: real 1.534s user 0.000s sys 0.000s W1010 20:58:55.165232 2864 rpcz_store.cc:238] Trace: 1010 20:58:51.930461 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:51.930469 (+ 8us) service_pool.cc:202] Handling call 1010 20:58:55.165198 (+3234729us) spinlock_profiling.cc:255] Waited 3.23 s on lock 0x3c2dc10. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f9e2c5b9181 00007f9e2b4a147c ffffffffffffffff 1010 20:58:55.165202 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7439707392} W1010 20:58:55.165246 2748 consensus_meta.cc:188] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 1.531s user 0.000s sys 0.000s I1010 20:58:55.165256 2861 raft_consensus.cc:1654] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 for term 2 because replica is already servicing an update from a current leader or another vote. I1010 20:58:55.165262 2748 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.668s W1010 20:58:55.165287 2861 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.6:47110 (request call id 3) took 3234ms (client timeout 1539). W1010 20:58:55.165314 2861 rpcz_store.cc:238] Trace: 1010 20:58:51.931008 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:51.931013 (+ 5us) service_pool.cc:202] Handling call 1010 20:58:55.165283 (+3234270us) spinlock_profiling.cc:255] Waited 3.23 s on lock 0x3c2dc10. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f9e2c5b9181 00007f9e2b4a147c ffffffffffffffff 1010 20:58:55.165286 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7438730496} I1010 20:58:55.165329 3722 leader_election.cc:223] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 2 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:58:55.165343 3722 leader_election.cc:223] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 2 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 W1010 20:58:55.165385 3819 consensus_meta.cc:188] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 2.435s user 0.000s sys 0.000s W1010 20:58:55.165385 3742 consensus_meta.cc:188] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 1.531s user 0.000s sys 0.000s I1010 20:58:55.165416 3819 raft_consensus.cc:1981] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.750s I1010 20:58:55.165437 3819 raft_consensus.cc:391] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } I1010 20:58:55.165472 3742 leader_election.cc:223] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 3 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:58:55.165503 3742 leader_election.cc:223] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 3 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 I1010 20:58:55.165648 2985 raft_consensus.cc:2031] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Advancing to term 3 W1010 20:58:55.165731 3881 consensus_meta.cc:188] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 2.896s user 0.000s sys 0.000s I1010 20:58:55.165753 3881 raft_consensus.cc:1981] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.523s I1010 20:58:55.165772 3881 raft_consensus.cc:391] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } W1010 20:58:55.165954 3837 consensus_meta.cc:188] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 1.533s user 0.000s sys 0.000s W1010 20:58:55.165983 3744 consensus_meta.cc:188] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 1.534s user 0.000s sys 0.000s W1010 20:58:55.165990 3713 consensus_meta.cc:188] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 1.529s user 0.000s sys 0.000s I1010 20:58:55.166076 3837 leader_election.cc:223] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 2 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:58:55.166108 3837 leader_election.cc:223] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 2 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 I1010 20:58:55.166165 3453 raft_consensus.cc:1584] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c for earlier term 2. Current term is 3. I1010 20:58:55.166193 3713 leader_election.cc:223] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 2 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 W1010 20:58:55.166211 3453 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.4:56561 (request call id 4) took 3235ms (client timeout 1735). W1010 20:58:55.166213 3220 consensus_meta.cc:188] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 3.232s user 0.000s sys 0.000s I1010 20:58:55.166219 3713 leader_election.cc:223] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 2 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 W1010 20:58:55.166225 3453 rpcz_store.cc:238] Trace: 1010 20:58:51.930631 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:51.930702 (+ 71us) service_pool.cc:202] Handling call 1010 20:58:55.166195 (+3235493us) spinlock_profiling.cc:255] Waited 3.24 s on lock 0x3e4f6d0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007fd760afa181 00007fd75f9e247c ffffffffffffffff 1010 20:58:55.166207 (+ 12us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7441532160} I1010 20:58:55.166241 3220 raft_consensus.cc:1981] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.763s I1010 20:58:55.166270 3219 raft_consensus.cc:1584] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 for earlier term 1. Current term is 2. W1010 20:58:55.166296 3818 consensus_meta.cc:188] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 4.107s user 0.000s sys 0.000s W1010 20:58:55.166302 3219 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.3:52871 (request call id 2) took 3237ms (client timeout 1569). I1010 20:58:55.166311 3818 raft_consensus.cc:1981] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.558s W1010 20:58:55.166316 3219 rpcz_store.cc:238] Trace: 1010 20:58:51.929127 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:51.929133 (+ 6us) service_pool.cc:202] Handling call 1010 20:58:55.166295 (+3237162us) spinlock_profiling.cc:255] Waited 3.24 s on lock 0x3b75970. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f8ce19fe181 00007f8ce08e647c ffffffffffffffff 1010 20:58:55.166299 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7445394816} I1010 20:58:55.166329 3818 raft_consensus.cc:391] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:58:55.166332 3100 raft_consensus.cc:1609] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 2: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. W1010 20:58:55.166386 3100 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.1:45293 (request call id 2) took 1533ms (client timeout 1762). W1010 20:58:55.166404 3100 rpcz_store.cc:238] Trace: 1010 20:58:53.633335 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:53.633343 (+ 8us) service_pool.cc:202] Handling call 1010 20:58:55.166377 (+1533034us) spinlock_profiling.cc:255] Waited 1.53 s on lock 0x4d75970. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007fc8aa54b181 00007fc8a943347c ffffffffffffffff 1010 20:58:55.166382 (+ 5us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3525841152} I1010 20:58:55.166476 2565 leader_election.cc:361] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 2 election: Vote denied by peer 33ef4317e50042ba95e517992caa6531. Message: Invalid argument: T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 2: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. I1010 20:58:55.166503 3744 leader_election.cc:223] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 2 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:58:55.166518 3744 leader_election.cc:223] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 2 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:58:55.166528 3332 raft_consensus.cc:1654] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 for term 1 because replica is already servicing an update from a current leader or another vote. W1010 20:58:55.166549 2509 consensus_meta.cc:188] T 4aa163baddb74c4998157964dd55b1f4 P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 1.532s user 0.000s sys 0.000s I1010 20:58:55.166566 2509 raft_consensus.cc:1677] T 4aa163baddb74c4998157964dd55b1f4 P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de6cfa864d64c58bf4eafb689517c93 in term 2. W1010 20:58:55.166573 3332 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.3:47101 (request call id 1) took 6494ms (client timeout 1511). W1010 20:58:55.166576 2509 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.3:38215 (request call id 4) took 1532ms (client timeout 1574). I1010 20:58:55.166574 3569 raft_consensus.cc:1981] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.673s W1010 20:58:55.166591 3332 rpcz_store.cc:238] Trace: 1010 20:58:48.671708 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:48.671718 (+ 10us) service_pool.cc:202] Handling call 1010 20:58:55.166564 (+6494846us) spinlock_profiling.cc:255] Waited 6.49 s on lock 0x3e5b6d0. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f3fb2fbf181 00007f3fb1ea747c ffffffffffffffff 1010 20:58:55.166568 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":14938013568} W1010 20:58:55.166591 2509 rpcz_store.cc:238] Trace: 1010 20:58:53.633836 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:53.633858 (+ 22us) service_pool.cc:202] Handling call 1010 20:58:55.166572 (+1532714us) inbound_call.cc:130] Queueing success response Metrics: {"tcmalloc_contention_cycles":3456} I1010 20:58:55.166613 3336 raft_consensus.cc:1609] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 2: Already voted for candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in this term. W1010 20:58:55.166635 3336 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.4:46735 (request call id 3) took 6497ms (client timeout 1675). W1010 20:58:55.166645 3336 rpcz_store.cc:238] Trace: 1010 20:58:48.669074 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:48.669100 (+ 26us) service_pool.cc:202] Handling call 1010 20:58:55.166629 (+6497529us) spinlock_profiling.cc:255] Waited 6.5 s on lock 0x3e5b6d0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f3fb2fbf181 00007f3fb1ea747c ffffffffffffffff 1010 20:58:55.166632 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":14944257408} I1010 20:58:55.166726 2746 raft_consensus.cc:1609] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 in current term 2: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:58:55.166790 3506 leader_election.cc:361] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 2 election: Vote denied by peer f6d15ccd64ec49be8396ed78e2a88377. Message: Invalid argument: T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 in current term 2: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:58:55.166812 3219 raft_consensus.cc:1609] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 2: Already voted for candidate 1380a27d6b824575a3a5e139bee139f4 in this term. W1010 20:58:55.166815 3692 consensus_meta.cc:188] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 5.070s user 0.000s sys 0.000s I1010 20:58:55.166828 3692 raft_consensus.cc:1981] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.749s I1010 20:58:55.166847 3692 raft_consensus.cc:391] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } I1010 20:58:55.166872 3570 raft_consensus.cc:1609] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 2: Already voted for candidate 580cbaf7092444dfbc6fb344da88d7b6 in this term. I1010 20:58:55.166873 2799 leader_election.cc:361] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 2 election: Vote denied by peer 1380a27d6b824575a3a5e139bee139f4. Message: Invalid argument: T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 2: Already voted for candidate 1380a27d6b824575a3a5e139bee139f4 in this term. I1010 20:58:55.166890 2628 raft_consensus.cc:1609] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 2: Already voted for candidate 1031512f5120422e9e22a0afabd78f61 in this term. W1010 20:58:55.166903 3570 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.4:54236 (request call id 3) took 1532ms (client timeout 1690). W1010 20:58:55.166916 3570 rpcz_store.cc:238] Trace: 1010 20:58:53.634611 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:53.634623 (+ 12us) service_pool.cc:202] Handling call 1010 20:58:55.166897 (+1532274us) spinlock_profiling.cc:255] Waited 1.53 s on lock 0x44ffc10. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f6cf36a4181 00007f6cf258c47c ffffffffffffffff 1010 20:58:55.166900 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3524116736} I1010 20:58:55.166934 3571 raft_consensus.cc:1654] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 for term 2 because replica is already servicing an update from a current leader or another vote. I1010 20:58:55.166965 2921 leader_election.cc:361] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: Vote denied by peer 580cbaf7092444dfbc6fb344da88d7b6. Message: Invalid argument: T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 2: Already voted for candidate 580cbaf7092444dfbc6fb344da88d7b6 in this term. I1010 20:58:55.166971 2921 leader_election.cc:248] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:55.166981 3037 leader_election.cc:361] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 2 election: Vote denied by peer 1031512f5120422e9e22a0afabd78f61. Message: Invalid argument: T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 2: Already voted for candidate 1031512f5120422e9e22a0afabd78f61 in this term. I1010 20:58:55.166990 2685 leader_election.cc:361] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 2 election: Vote denied by peer 580cbaf7092444dfbc6fb344da88d7b6. Message: Service unavailable: T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 for term 2 because replica is already servicing an update from a current leader or another vote. I1010 20:58:55.166998 2685 leader_election.cc:248] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. W1010 20:58:55.167104 3678 consensus_meta.cc:188] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 1.530s user 0.000s sys 0.000s I1010 20:58:55.167155 3452 raft_consensus.cc:1654] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 for term 2 because replica is already servicing an update from a current leader or another vote. I1010 20:58:55.167176 3678 leader_election.cc:223] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 2 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 W1010 20:58:55.167187 3452 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.7:59528 (request call id 2) took 1533ms (client timeout 1503). I1010 20:58:55.167192 3678 leader_election.cc:223] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 2 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 W1010 20:58:55.167201 3452 rpcz_store.cc:238] Trace: 1010 20:58:53.633620 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:53.634867 (+ 1247us) service_pool.cc:202] Handling call 1010 20:58:55.167180 (+1532313us) spinlock_profiling.cc:255] Waited 1.53 s on lock 0x3e4f6d0. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007fd760afa181 00007fd75f9e247c ffffffffffffffff 1010 20:58:55.167183 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3524231936} I1010 20:58:55.167203 4113 raft_consensus.cc:1981] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.663s I1010 20:58:55.167212 4113 raft_consensus.cc:1823] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given I1010 20:58:55.167222 4071 raft_consensus.cc:1981] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.823s I1010 20:58:55.167235 4071 raft_consensus.cc:1823] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: None given I1010 20:58:55.167315 3099 raft_consensus.cc:1654] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 for term 3 because replica is already servicing an update from a current leader or another vote. I1010 20:58:55.167346 3102 raft_consensus.cc:1609] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 in current term 2: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. W1010 20:58:55.167352 3099 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.6:59112 (request call id 3) took 1533ms (client timeout 1584). W1010 20:58:55.167353 3101 consensus_meta.cc:188] T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 1.531s user 0.000s sys 0.000s I1010 20:58:55.167366 3101 raft_consensus.cc:1677] T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 580cbaf7092444dfbc6fb344da88d7b6 in term 2. W1010 20:58:55.167373 3099 rpcz_store.cc:238] Trace: 1010 20:58:53.633587 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:53.633597 (+ 10us) service_pool.cc:202] Handling call 1010 20:58:55.167345 (+1533748us) spinlock_profiling.cc:255] Waited 1.53 s on lock 0x4d756d0. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007fc8aa54b181 00007fc8a943347c ffffffffffffffff 1010 20:58:55.167348 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3527527168} I1010 20:58:55.167373 2802 leader_election.cc:347] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 2 election: Vote granted by peer 84bde34331b94c4ea003e48f9317f792 W1010 20:58:55.167379 3101 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.9:54865 (request call id 2) took 1534ms (client timeout 1681). I1010 20:58:55.167381 2802 leader_election.cc:248] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. W1010 20:58:55.167387 3102 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.6:59112 (request call id 2) took 4771ms (client timeout 1650). W1010 20:58:55.167392 3101 rpcz_store.cc:238] Trace: 1010 20:58:53.632628 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:53.632664 (+ 36us) service_pool.cc:202] Handling call 1010 20:58:55.167373 (+1534709us) inbound_call.cc:130] Queueing success response Metrics: {"tcmalloc_contention_cycles":1017088} W1010 20:58:55.167400 3102 rpcz_store.cc:238] Trace: 1010 20:58:50.395423 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:50.395438 (+ 15us) service_pool.cc:202] Handling call 1010 20:58:55.167379 (+4771941us) spinlock_profiling.cc:255] Waited 4.77 s on lock 0x4d756d0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007fc8aa54b181 00007fc8a943347c ffffffffffffffff 1010 20:58:55.167383 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":10975344896} I1010 20:58:55.167420 3098 raft_consensus.cc:1654] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 for term 2 because replica is already servicing an update from a current leader or another vote. W1010 20:58:55.167445 3098 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.0:46745 (request call id 4) took 1533ms (client timeout 1521). I1010 20:58:55.167450 2984 raft_consensus.cc:1609] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 in current term 2: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. W1010 20:58:55.167461 3098 rpcz_store.cc:238] Trace: 1010 20:58:53.634128 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:53.634136 (+ 8us) service_pool.cc:202] Handling call 1010 20:58:55.167438 (+1533302us) spinlock_profiling.cc:255] Waited 1.53 s on lock 0x4d756d0. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007fc8aa54b181 00007fc8a943347c ffffffffffffffff 1010 20:58:55.167440 (+ 2us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3526518656,"tcmalloc_contention_cycles":3328} I1010 20:58:55.167459 3153 leader_election.cc:361] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 3 election: Vote denied by peer 33ef4317e50042ba95e517992caa6531. Message: Service unavailable: T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 for term 3 because replica is already servicing an update from a current leader or another vote. I1010 20:58:55.167471 3153 leader_election.cc:248] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I1010 20:58:55.167502 3507 leader_election.cc:361] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 2 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c. Message: Invalid argument: T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 in current term 2: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. W1010 20:58:55.167505 3842 consensus_meta.cc:188] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 1.428s user 0.000s sys 0.000s I1010 20:58:55.167510 3507 leader_election.cc:248] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:55.167528 3842 raft_consensus.cc:1981] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.513s I1010 20:58:55.167549 3842 raft_consensus.cc:391] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:58:55.167582 3219 raft_consensus.cc:1584] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 for earlier term 2. Current term is 3. W1010 20:58:55.167651 3036 leader_election.cc:333] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 2 election: Vote denied by peer 1380a27d6b824575a3a5e139bee139f4 with higher term. Message: Invalid argument: T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 for earlier term 2. Current term is 3. I1010 20:58:55.167660 3036 leader_election.cc:336] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term I1010 20:58:55.167692 3504 leader_election.cc:347] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 2 election: Vote granted by peer 33ef4317e50042ba95e517992caa6531 I1010 20:58:55.167701 3504 leader_election.cc:248] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I1010 20:58:55.167747 4114 raft_consensus.cc:1981] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.722s I1010 20:58:55.167748 2984 raft_consensus.cc:1609] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 2: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:58:55.167753 2509 raft_consensus.cc:1609] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 2: Already voted for candidate 84bde34331b94c4ea003e48f9317f792 in this term. I1010 20:58:55.167757 4114 raft_consensus.cc:1823] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given I1010 20:58:55.167800 4115 raft_consensus.cc:1981] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.777s I1010 20:58:55.167826 4117 raft_consensus.cc:1981] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.733s I1010 20:58:55.167839 4117 raft_consensus.cc:1823] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given I1010 20:58:55.167837 3039 leader_election.cc:361] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 2 election: Vote denied by peer 84bde34331b94c4ea003e48f9317f792. Message: Invalid argument: T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 2: Already voted for candidate 84bde34331b94c4ea003e48f9317f792 in this term. I1010 20:58:55.167857 4118 raft_consensus.cc:1981] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.669s I1010 20:58:55.167867 4118 raft_consensus.cc:1823] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 1380a27d6b824575a3a5e139bee139f4 with higher term. Message: Invalid argument: T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 for earlier term 2. Current term is 3. I1010 20:58:55.167812 2802 leader_election.cc:361] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 2 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c. Message: Invalid argument: T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 2: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:58:55.167886 4115 raft_consensus.cc:1861] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Leader election won for term 2 I1010 20:58:55.167937 2802 leader_election.cc:248] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:55.168015 4119 raft_consensus.cc:1981] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.597s I1010 20:58:55.168027 4119 raft_consensus.cc:1861] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Leader election won for term 2 I1010 20:58:55.168092 4121 raft_consensus.cc:1981] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.500s I1010 20:58:55.168107 4121 raft_consensus.cc:1823] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given I1010 20:58:55.168105 4116 raft_consensus.cc:1981] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.741s I1010 20:58:55.168117 4116 raft_consensus.cc:1823] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given I1010 20:58:55.168094 4115 raft_consensus.cc:467] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 LEADER]: Becoming Leader. State: Replica: 8de6cfa864d64c58bf4eafb689517c93, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:55.168182 4115 consensus_queue.cc:151] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 2, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } I1010 20:58:55.168421 4119 raft_consensus.cc:467] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 LEADER]: Becoming Leader. State: Replica: 580cbaf7092444dfbc6fb344da88d7b6, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:55.168520 4119 consensus_queue.cc:151] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 2, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:58:55.170699 2409 catalog_manager.cc:2018] T ed24ca4817744964991178277119b367 reported consensus state change: term changed from 0 to 2, leader changed from to 580cbaf7092444dfbc6fb344da88d7b6 (127.6.150.9). New consensus state: current_term: 2 leader_uuid: "580cbaf7092444dfbc6fb344da88d7b6" config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } } I1010 20:58:55.170773 2403 catalog_manager.cc:2018] T 4aa163baddb74c4998157964dd55b1f4 reported consensus state change: term changed from 0 to 2, leader changed from to 8de6cfa864d64c58bf4eafb689517c93 (127.6.150.3). New consensus state: current_term: 2 leader_uuid: "8de6cfa864d64c58bf4eafb689517c93" config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } } I1010 20:58:55.199079 3715 raft_consensus.cc:375] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:55.199090 3715 raft_consensus.cc:2031] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Advancing to term 3 W1010 20:58:55.208984 2800 leader_election.cc:275] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1031512f5120422e9e22a0afabd78f61: Timed out: RequestConsensusVote RPC to 127.6.150.1:46963 timed out after 1.574s I1010 20:58:55.226748 3781 raft_consensus.cc:375] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:55.226776 3781 raft_consensus.cc:2031] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Advancing to term 3 W1010 20:58:55.314004 3507 leader_election.cc:275] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7f623b00f5ae4b9c8e8940692d5dfd44: Timed out: RequestConsensusVote RPC to 127.6.150.7:45436 timed out after 1.681s W1010 20:58:55.339257 3039 leader_election.cc:275] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 84bde34331b94c4ea003e48f9317f792: Timed out: RequestConsensusVote RPC to 127.6.150.0:40113 timed out after 1.703s I1010 20:58:55.339272 3039 leader_election.cc:248] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:55.339399 4131 raft_consensus.cc:1981] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.531s I1010 20:58:55.339414 4131 raft_consensus.cc:1823] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given W1010 20:58:55.355604 2568 leader_election.cc:275] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7f623b00f5ae4b9c8e8940692d5dfd44: Timed out: RequestConsensusVote RPC to 127.6.150.7:45436 timed out after 1.720s W1010 20:58:55.355628 2568 leader_election.cc:275] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 84bde34331b94c4ea003e48f9317f792: Timed out: RequestConsensusVote RPC to 127.6.150.0:40113 timed out after 1.720s I1010 20:58:55.355630 2568 leader_election.cc:248] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. W1010 20:58:55.392796 3389 leader_election.cc:275] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer f6d15ccd64ec49be8396ed78e2a88377: Timed out: RequestConsensusVote RPC to 127.6.150.2:33554 timed out after 1.757s W1010 20:58:55.392905 3387 leader_election.cc:275] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1380a27d6b824575a3a5e139bee139f4: Timed out: RequestConsensusVote RPC to 127.6.150.6:46774 timed out after 1.757s I1010 20:58:55.392921 3387 leader_election.cc:248] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I1010 20:58:55.393097 4133 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.906s I1010 20:58:55.393131 4133 raft_consensus.cc:1823] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given W1010 20:58:55.396500 2567 leader_election.cc:275] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f6d15ccd64ec49be8396ed78e2a88377: Timed out: RequestConsensusVote RPC to 127.6.150.2:33554 timed out after 1.762s I1010 20:58:55.396530 2567 leader_election.cc:248] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:55.396697 4134 raft_consensus.cc:1981] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.525s I1010 20:58:55.396710 4134 raft_consensus.cc:1823] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given W1010 20:58:55.414990 2682 leader_election.cc:275] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1380a27d6b824575a3a5e139bee139f4: Timed out: RequestConsensusVote RPC to 127.6.150.6:46774 timed out after 1.780s I1010 20:58:55.415005 2682 leader_election.cc:248] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:55.415213 4135 raft_consensus.cc:1981] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.634s I1010 20:58:55.415240 4135 raft_consensus.cc:1823] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given I1010 20:58:55.461853 3808 raft_consensus.cc:375] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:55.461865 3808 raft_consensus.cc:2031] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:58:55.668601 2509 raft_consensus.cc:857] T 4aa163baddb74c4998157964dd55b1f4 P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Refusing update from remote peer 8de6cfa864d64c58bf4eafb689517c93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch) I1010 20:58:55.668736 4115 consensus_queue.cc:652] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 [LEADER]: Connected to new peer: Peer: 84bde34331b94c4ea003e48f9317f792, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:58:55.668829 3098 raft_consensus.cc:857] T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Refusing update from remote peer 580cbaf7092444dfbc6fb344da88d7b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch) I1010 20:58:55.669003 4137 consensus_queue.cc:652] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [LEADER]: Connected to new peer: Peer: 33ef4317e50042ba95e517992caa6531, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false W1010 20:58:56.247727 3869 consensus_meta.cc:188] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.083s user 0.000s sys 0.000s W1010 20:58:56.247727 2630 consensus_meta.cc:188] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 1.083s user 0.000s sys 0.000s W1010 20:58:56.247728 3671 consensus_meta.cc:188] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 1.083s user 0.000s sys 0.000s I1010 20:58:56.247779 2630 raft_consensus.cc:1677] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8062d66fd284035ae3bc2567bc11e7c in term 2. W1010 20:58:56.247824 2630 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.4:41994 (request call id 2) took 5849ms (client timeout 1726). W1010 20:58:56.247854 2630 rpcz_store.cc:238] Trace: 1010 20:58:50.398281 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:50.398288 (+ 7us) service_pool.cc:202] Handling call 1010 20:58:56.247806 (+5849518us) inbound_call.cc:130] Queueing success response Metrics: {"tcmalloc_contention_cycles":436096} W1010 20:58:56.247881 3881 consensus_meta.cc:188] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 1.082s user 0.000s sys 0.000s W1010 20:58:56.247894 3818 consensus_meta.cc:188] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 1.082s user 0.000s sys 0.000s W1010 20:58:56.247913 3692 consensus_meta.cc:188] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 1.081s user 0.000s sys 0.000s W1010 20:58:56.247946 3569 consensus_meta.cc:188] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6: Time spent flushing consensus metadata: real 1.081s user 0.000s sys 0.000s I1010 20:58:56.247962 3569 raft_consensus.cc:1677] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8062d66fd284035ae3bc2567bc11e7c in term 2. I1010 20:58:56.247975 3569 rpcz_store.cc:248] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.4:54236 (request call id 4) took 1087ms. Request Metrics: {} W1010 20:58:56.248034 3853 consensus_meta.cc:188] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 4.417s user 0.000s sys 0.000s I1010 20:58:56.248054 3853 raft_consensus.cc:1981] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.746s I1010 20:58:56.248103 3853 raft_consensus.cc:391] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } W1010 20:58:56.248105 2985 consensus_meta.cc:188] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.082s user 0.000s sys 0.000s I1010 20:58:56.248119 2985 raft_consensus.cc:1981] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.757s W1010 20:58:56.248280 3817 consensus_meta.cc:188] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 4.259s user 0.000s sys 0.000s I1010 20:58:56.248297 3817 raft_consensus.cc:1981] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.672s W1010 20:58:56.248324 3819 consensus_meta.cc:188] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 1.083s user 0.000s sys 0.000s W1010 20:58:56.248342 3755 consensus_meta.cc:188] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 4.337s user 0.000s sys 0.000s I1010 20:58:56.248361 3755 raft_consensus.cc:1981] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.506s I1010 20:58:56.248414 3755 raft_consensus.cc:391] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } W1010 20:58:56.248528 3863 consensus_meta.cc:188] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 4.028s user 0.000s sys 0.000s I1010 20:58:56.248541 3863 raft_consensus.cc:1981] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.754s W1010 20:58:56.248553 2631 consensus_meta.cc:188] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 2.615s user 0.000s sys 0.000s I1010 20:58:56.248560 3863 raft_consensus.cc:391] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } I1010 20:58:56.248566 2631 raft_consensus.cc:1981] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.667s W1010 20:58:56.248579 3729 consensus_meta.cc:188] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 1.084s user 0.000s sys 0.000s W1010 20:58:56.248692 3834 consensus_meta.cc:188] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 1.084s user 0.000s sys 0.000s I1010 20:58:56.248767 3817 raft_consensus.cc:391] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:58:56.248796 3867 raft_consensus.cc:375] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:56.248805 3867 raft_consensus.cc:2031] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Advancing to term 3 I1010 20:58:56.248865 2921 leader_election.cc:347] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: Vote granted by peer 580cbaf7092444dfbc6fb344da88d7b6 W1010 20:58:56.248864 3717 consensus_meta.cc:188] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 7.499s user 0.000s sys 0.000s W1010 20:58:56.248870 2748 consensus_meta.cc:188] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 1.084s user 0.000s sys 0.000s I1010 20:58:56.248875 2921 leader_election.cc:248] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I1010 20:58:56.248880 2748 raft_consensus.cc:1677] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42e94fc3adbb419591836d222d946978 in term 3. W1010 20:58:56.248890 2748 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.8:42100 (request call id 5) took 2614ms (client timeout 1757). W1010 20:58:56.248900 2748 rpcz_store.cc:238] Trace: 1010 20:58:53.634703 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:53.634714 (+ 11us) service_pool.cc:202] Handling call 1010 20:58:56.248886 (+2614172us) inbound_call.cc:130] Queueing success response Metrics: {} W1010 20:58:56.249024 3842 consensus_meta.cc:188] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 1.081s user 0.000s sys 0.000s I1010 20:58:56.249027 4144 raft_consensus.cc:1981] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.683s I1010 20:58:56.249043 4144 raft_consensus.cc:1861] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Leader election won for term 2 I1010 20:58:56.249177 4144 raft_consensus.cc:467] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [term 2 LEADER]: Becoming Leader. State: Replica: b8062d66fd284035ae3bc2567bc11e7c, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:58:56.249207 4144 consensus_queue.cc:151] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 2, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } W1010 20:58:56.249265 3808 consensus_meta.cc:188] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 0.787s user 0.000s sys 0.000s I1010 20:58:56.249277 3808 raft_consensus.cc:1981] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.650s I1010 20:58:56.249296 3808 raft_consensus.cc:391] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } W1010 20:58:56.249449 3220 consensus_meta.cc:188] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 1.083s user 0.000s sys 0.000s I1010 20:58:56.249467 3220 raft_consensus.cc:1677] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42e94fc3adbb419591836d222d946978 in term 2. W1010 20:58:56.249486 3220 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.8:57074 (request call id 3) took 4315ms (client timeout 1605). I1010 20:58:56.249491 3859 raft_consensus.cc:375] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:56.249503 3859 raft_consensus.cc:2031] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Advancing to term 3 W1010 20:58:56.249497 3220 rpcz_store.cc:238] Trace: 1010 20:58:51.934387 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:51.934397 (+ 10us) service_pool.cc:202] Handling call 1010 20:58:56.249479 (+4315082us) inbound_call.cc:130] Queueing success response Metrics: {} W1010 20:58:56.249743 3217 consensus_meta.cc:188] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 2.615s user 0.000s sys 0.000s I1010 20:58:56.249763 3217 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.529s I1010 20:58:56.249800 3859 raft_consensus.cc:1981] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.714s I1010 20:58:56.249825 3859 raft_consensus.cc:391] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } W1010 20:58:56.249877 3794 consensus_meta.cc:188] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 4.176s user 0.000s sys 0.000s I1010 20:58:56.249892 3794 raft_consensus.cc:1981] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.674s I1010 20:58:56.249910 3794 raft_consensus.cc:391] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } I1010 20:58:56.250154 3717 raft_consensus.cc:1981] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.567s I1010 20:58:56.250174 3717 raft_consensus.cc:391] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } I1010 20:58:56.250423 3869 leader_election.cc:223] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 3 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 I1010 20:58:56.250442 3869 leader_election.cc:223] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 3 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:58:56.250455 3881 leader_election.cc:223] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 2 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:58:56.250460 2986 raft_consensus.cc:1584] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 for earlier term 2. Current term is 3. I1010 20:58:56.250495 3881 leader_election.cc:223] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 2 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:58:56.250532 2986 rpcz_store.cc:248] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.7:55183 (request call id 4) took 1083ms. Request Metrics: {"spinlock_wait_cycles":2491646720} I1010 20:58:56.250586 3332 raft_consensus.cc:2031] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Advancing to term 3 W1010 20:58:56.250605 3273 leader_election.cc:333] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 2 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c with higher term. Message: Invalid argument: T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 for earlier term 2. Current term is 3. I1010 20:58:56.250613 3273 leader_election.cc:336] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term I1010 20:58:56.250643 3692 leader_election.cc:223] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 2 election: Requesting vote from peer 580cbaf7092444dfbc6fb344da88d7b6 I1010 20:58:56.250661 3692 leader_election.cc:223] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 2 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:58:56.250705 4048 raft_consensus.cc:1981] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.601s I1010 20:58:56.250746 4048 raft_consensus.cc:1823] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Leader election lost for term 1. Reason: None given I1010 20:58:56.250759 3569 raft_consensus.cc:1609] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 in current term 2: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:58:56.250779 2507 raft_consensus.cc:1654] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c for term 2 because replica is already servicing an update from a current leader or another vote. I1010 20:58:56.250828 2514 raft_consensus.cc:1584] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 for earlier term 1. Current term is 2. I1010 20:58:56.250856 2507 rpcz_store.cc:248] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.4:40901 (request call id 4) took 1090ms. Request Metrics: {"spinlock_wait_cycles":2508205824} I1010 20:58:56.250856 2450 leader_election.cc:361] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 2 election: Vote denied by peer 580cbaf7092444dfbc6fb344da88d7b6. Message: Invalid argument: T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 in current term 2: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. W1010 20:58:56.250861 2514 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.9:37963 (request call id 1) took 5854ms (client timeout 1576). W1010 20:58:56.250875 2514 rpcz_store.cc:238] Trace: 1010 20:58:50.395960 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:50.395967 (+ 7us) service_pool.cc:202] Handling call 1010 20:58:56.250855 (+5854888us) spinlock_profiling.cc:255] Waited 5.85 s on lock 0x409b970. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007ff4d62cf181 00007ff4d51b747c ffffffffffffffff 1010 20:58:56.250858 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":13466062080} I1010 20:58:56.250928 3671 leader_election.cc:223] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 2 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:58:56.250932 2921 leader_election.cc:361] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 2 election: Vote denied by peer 84bde34331b94c4ea003e48f9317f792. Message: Service unavailable: T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c for term 2 because replica is already servicing an update from a current leader or another vote. I1010 20:58:56.250952 3671 leader_election.cc:223] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 2 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:58:56.251039 3452 raft_consensus.cc:1609] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 2: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:58:56.251106 2801 leader_election.cc:361] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 2 election: Vote denied by peer 42e94fc3adbb419591836d222d946978. Message: Invalid argument: T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 2: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:58:56.251126 2863 raft_consensus.cc:1609] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 2: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. I1010 20:58:56.251189 3390 leader_election.cc:361] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 2 election: Vote denied by peer 8de6cfa864d64c58bf4eafb689517c93. Message: Invalid argument: T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 2: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. I1010 20:58:56.251332 3818 leader_election.cc:223] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 2 election: Requesting vote from peer 580cbaf7092444dfbc6fb344da88d7b6 I1010 20:58:56.251351 3818 leader_election.cc:223] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 2 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:58:56.251415 4088 raft_consensus.cc:1981] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.794s I1010 20:58:56.251420 3842 leader_election.cc:223] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 2 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:58:56.251446 3842 leader_election.cc:223] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 2 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 I1010 20:58:56.251446 3098 raft_consensus.cc:1640] T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 for term 2 because replica is either leader or believes a valid leader to be alive. I1010 20:58:56.251452 3569 raft_consensus.cc:1640] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 LEADER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 for term 2 because replica is either leader or believes a valid leader to be alive. I1010 20:58:56.251459 4088 raft_consensus.cc:1823] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Leader election lost for term 1. Reason: None given I1010 20:58:56.251472 3333 raft_consensus.cc:1609] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 in current term 2: Already voted for candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in this term. W1010 20:58:56.251497 3333 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.9:42466 (request call id 4) took 2618ms (client timeout 1681). I1010 20:58:56.251502 3270 leader_election.cc:361] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 2 election: Vote denied by peer 33ef4317e50042ba95e517992caa6531. Message: Invalid argument: T ed24ca4817744964991178277119b367 P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 for term 2 because replica is either leader or believes a valid leader to be alive. W1010 20:58:56.251528 3333 rpcz_store.cc:238] Trace: 1010 20:58:53.632568 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:53.632584 (+ 16us) service_pool.cc:202] Handling call 1010 20:58:56.251489 (+2618905us) spinlock_profiling.cc:255] Waited 2.62 s on lock 0x3e5bc10. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f3fb2fbf181 00007f3fb1ea747c ffffffffffffffff 1010 20:58:56.251493 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":6023385472} I1010 20:58:56.251550 3273 leader_election.cc:361] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 2 election: Vote denied by peer 580cbaf7092444dfbc6fb344da88d7b6. Message: Invalid argument: T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 LEADER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 for term 2 because replica is either leader or believes a valid leader to be alive. I1010 20:58:56.251559 3273 leader_election.cc:248] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:56.251569 3336 raft_consensus.cc:857] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Refusing update from remote peer 580cbaf7092444dfbc6fb344da88d7b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch) I1010 20:58:56.251585 4088 raft_consensus.cc:1981] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.771s I1010 20:58:56.251588 4088 raft_consensus.cc:1823] T ed24ca4817744964991178277119b367 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given I1010 20:58:56.251613 3834 leader_election.cc:223] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 2 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:58:56.251638 3834 leader_election.cc:223] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 2 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 I1010 20:58:56.251708 4119 consensus_queue.cc:652] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 [LEADER]: Connected to new peer: Peer: 7f623b00f5ae4b9c8e8940692d5dfd44, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:58:56.251759 3098 raft_consensus.cc:1609] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 2: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. I1010 20:58:56.251830 2682 leader_election.cc:361] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 2 election: Vote denied by peer 33ef4317e50042ba95e517992caa6531. Message: Invalid argument: T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 2: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. I1010 20:58:56.251850 2747 raft_consensus.cc:1609] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 2: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:58:56.251938 3819 leader_election.cc:223] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 2 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:58:56.251950 3819 leader_election.cc:223] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 2 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 I1010 20:58:56.251971 3038 leader_election.cc:361] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 2 election: Vote denied by peer f6d15ccd64ec49be8396ed78e2a88377. Message: Invalid argument: T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 2: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:58:56.251986 4105 raft_consensus.cc:1981] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.730s I1010 20:58:56.252109 2630 raft_consensus.cc:1609] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 2: Already voted for candidate 1031512f5120422e9e22a0afabd78f61 in this term. I1010 20:58:56.252140 3098 raft_consensus.cc:1609] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 2: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. I1010 20:58:56.252183 2682 leader_election.cc:361] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 2 election: Vote denied by peer 33ef4317e50042ba95e517992caa6531. Message: Invalid argument: T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 2: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. I1010 20:58:56.252194 2683 leader_election.cc:361] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 2 election: Vote denied by peer 1031512f5120422e9e22a0afabd78f61. Message: Invalid argument: T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 2: Already voted for candidate 1031512f5120422e9e22a0afabd78f61 in this term. I1010 20:58:56.252207 2683 leader_election.cc:248] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:56.252212 3729 leader_election.cc:223] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 2 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:58:56.252220 3729 leader_election.cc:223] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 2 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:58:56.252251 2745 raft_consensus.cc:1609] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 2: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:58:56.252269 2749 raft_consensus.cc:1654] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 for term 2 because replica is already servicing an update from a current leader or another vote. W1010 20:58:56.252271 2745 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.1:40603 (request call id 2) took 2618ms (client timeout 1762). W1010 20:58:56.252297 2745 rpcz_store.cc:238] Trace: 1010 20:58:53.633387 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:53.633394 (+ 7us) service_pool.cc:202] Handling call 1010 20:58:56.252265 (+2618871us) spinlock_profiling.cc:255] Waited 2.62 s on lock 0x37cc1d0. stack: 1010 20:58:56.252268 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":6022850944} I1010 20:58:56.252310 2749 rpcz_store.cc:248] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.5:40398 (request call id 3) took 1085ms. Request Metrics: {"spinlock_wait_cycles":2496500480} I1010 20:58:56.252328 2986 raft_consensus.cc:1640] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [term 2 LEADER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 for term 2 because replica is either leader or believes a valid leader to be alive. I1010 20:58:56.252354 2863 raft_consensus.cc:1609] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 2: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. I1010 20:58:56.252364 3038 leader_election.cc:361] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 2 election: Vote denied by peer f6d15ccd64ec49be8396ed78e2a88377. Message: Service unavailable: T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 for term 2 because replica is already servicing an update from a current leader or another vote. I1010 20:58:56.252385 3038 leader_election.cc:248] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:56.252442 2748 raft_consensus.cc:1609] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 2: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:58:56.252485 2450 leader_election.cc:361] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 2 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c. Message: Invalid argument: T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [term 2 LEADER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 for term 2 because replica is either leader or believes a valid leader to be alive. I1010 20:58:56.252501 2450 leader_election.cc:248] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:56.252523 2685 leader_election.cc:361] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 2 election: Vote denied by peer 8de6cfa864d64c58bf4eafb689517c93. Message: Invalid argument: T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 2: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. I1010 20:58:56.252542 4048 raft_consensus.cc:1981] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.504s I1010 20:58:56.252547 4048 raft_consensus.cc:1823] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given I1010 20:58:56.252564 3389 leader_election.cc:361] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 2 election: Vote denied by peer f6d15ccd64ec49be8396ed78e2a88377. Message: Invalid argument: T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 2: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:58:56.252573 4105 raft_consensus.cc:1823] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Leader election lost for term 1. Reason: None given I1010 20:58:56.252580 3389 leader_election.cc:248] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:56.252655 3452 raft_consensus.cc:1609] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 2: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:58:56.252667 2403 catalog_manager.cc:2018] T 970154c84d6849cc83af1d9e5ad6980d reported consensus state change: term changed from 0 to 2, leader changed from to b8062d66fd284035ae3bc2567bc11e7c (127.6.150.4). New consensus state: current_term: 2 leader_uuid: "b8062d66fd284035ae3bc2567bc11e7c" config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } } I1010 20:58:56.252769 2684 leader_election.cc:361] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 2 election: Vote denied by peer 42e94fc3adbb419591836d222d946978. Message: Invalid argument: T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 2: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:58:56.252781 2684 leader_election.cc:248] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:56.253473 2746 raft_consensus.cc:1654] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 for term 2 because replica is already servicing an update from a current leader or another vote. I1010 20:58:56.253584 2801 leader_election.cc:361] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 2 election: Vote denied by peer f6d15ccd64ec49be8396ed78e2a88377. Message: Service unavailable: T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 for term 2 because replica is already servicing an update from a current leader or another vote. I1010 20:58:56.253603 2801 leader_election.cc:248] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:56.253706 4156 raft_consensus.cc:1981] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.705s I1010 20:58:56.253720 4151 raft_consensus.cc:1981] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.543s I1010 20:58:56.253741 4156 raft_consensus.cc:1823] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given I1010 20:58:56.253748 4105 raft_consensus.cc:1981] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.508s I1010 20:58:56.253753 4151 raft_consensus.cc:1823] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given I1010 20:58:56.253758 4105 raft_consensus.cc:1823] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given I1010 20:58:56.253793 4153 raft_consensus.cc:1981] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.717s I1010 20:58:56.253829 4150 raft_consensus.cc:1981] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.650s I1010 20:58:56.253844 4150 raft_consensus.cc:1823] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given I1010 20:58:56.253855 4153 raft_consensus.cc:1823] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given W1010 20:58:56.330180 2919 consensus_peers.cc:332] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 84fcfff9fdef4acf8c2b47f36330567f. Status: Timed out: UpdateConsensus RPC to 127.6.150.1:46963 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 3 times. W1010 20:58:56.670840 2800 consensus_peers.cc:332] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 4aa163baddb74c4998157964dd55b1f4. Status: Timed out: UpdateConsensus RPC to 127.6.150.1:46963 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 1 times. W1010 20:58:56.702596 3273 leader_election.cc:275] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8de6cfa864d64c58bf4eafb689517c93: Timed out: RequestConsensusVote RPC to 127.6.150.3:40117 timed out after 1.534s W1010 20:58:56.725730 3507 leader_election.cc:275] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8de6cfa864d64c58bf4eafb689517c93: Timed out: RequestConsensusVote RPC to 127.6.150.3:40117 timed out after 1.564s I1010 20:58:56.725772 3507 leader_election.cc:248] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:56.725982 4159 raft_consensus.cc:1981] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.788s I1010 20:58:56.725999 4159 raft_consensus.cc:1823] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given I1010 20:58:56.752178 3569 raft_consensus.cc:857] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Refusing update from remote peer b8062d66fd284035ae3bc2567bc11e7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch) I1010 20:58:56.752255 2514 raft_consensus.cc:857] T 970154c84d6849cc83af1d9e5ad6980d P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Refusing update from remote peer b8062d66fd284035ae3bc2567bc11e7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch) I1010 20:58:56.752347 4144 consensus_queue.cc:652] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [LEADER]: Connected to new peer: Peer: 580cbaf7092444dfbc6fb344da88d7b6, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:58:56.752506 4160 consensus_queue.cc:652] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c [LEADER]: Connected to new peer: Peer: 84bde34331b94c4ea003e48f9317f792, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false W1010 20:58:56.758087 3156 leader_election.cc:275] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8de6cfa864d64c58bf4eafb689517c93: Timed out: RequestConsensusVote RPC to 127.6.150.3:40117 timed out after 1.591s W1010 20:58:56.758832 3154 leader_election.cc:275] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1031512f5120422e9e22a0afabd78f61: Timed out: RequestConsensusVote RPC to 127.6.150.1:46963 timed out after 1.591s I1010 20:58:56.758846 3154 leader_election.cc:248] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:56.758970 4164 raft_consensus.cc:1981] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.522s I1010 20:58:56.758985 4164 raft_consensus.cc:1823] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given W1010 20:58:56.780542 3390 leader_election.cc:275] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer b8062d66fd284035ae3bc2567bc11e7c: Timed out: RequestConsensusVote RPC to 127.6.150.4:44055 timed out after 1.614s W1010 20:58:56.780586 3390 leader_election.cc:275] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 7f623b00f5ae4b9c8e8940692d5dfd44: Timed out: RequestConsensusVote RPC to 127.6.150.7:45436 timed out after 1.614s I1010 20:58:56.780593 3390 leader_election.cc:248] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I1010 20:58:56.780773 4165 raft_consensus.cc:1981] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.520s I1010 20:58:56.780787 4165 raft_consensus.cc:1823] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given I1010 20:58:56.804744 3882 raft_consensus.cc:375] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:56.804756 3882 raft_consensus.cc:2031] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Advancing to term 3 I1010 20:58:56.806427 3891 raft_consensus.cc:375] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:56.806439 3891 raft_consensus.cc:2031] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Advancing to term 3 I1010 20:58:56.806779 3891 raft_consensus.cc:1981] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.641s I1010 20:58:56.806844 3891 raft_consensus.cc:391] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } I1010 20:58:56.825356 3754 raft_consensus.cc:375] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:56.825397 3754 raft_consensus.cc:2031] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Advancing to term 3 I1010 20:58:56.879410 3673 raft_consensus.cc:375] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:56.879426 3673 raft_consensus.cc:2031] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Advancing to term 3 I1010 20:58:56.879685 3673 raft_consensus.cc:1981] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.513s I1010 20:58:56.879721 3673 raft_consensus.cc:391] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } I1010 20:58:56.937952 3871 raft_consensus.cc:375] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:56.937968 3871 raft_consensus.cc:2031] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Advancing to term 3 I1010 20:58:57.123172 3829 raft_consensus.cc:375] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:57.123212 3829 raft_consensus.cc:2031] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Advancing to term 3 I1010 20:58:57.269773 3789 raft_consensus.cc:375] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:57.269791 3789 raft_consensus.cc:2031] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Advancing to term 3 I1010 20:58:57.373584 3805 raft_consensus.cc:375] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:57.373600 3805 raft_consensus.cc:2031] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Advancing to term 3 W1010 20:58:57.765588 3039 leader_election.cc:275] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7f623b00f5ae4b9c8e8940692d5dfd44: Timed out: RequestConsensusVote RPC to 127.6.150.7:45436 timed out after 1.513s I1010 20:58:57.765609 3039 leader_election.cc:248] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:57.765851 4166 raft_consensus.cc:1981] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.708s I1010 20:58:57.765867 4166 raft_consensus.cc:1823] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given W1010 20:58:57.825301 2685 leader_election.cc:275] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7f623b00f5ae4b9c8e8940692d5dfd44: Timed out: RequestConsensusVote RPC to 127.6.150.7:45436 timed out after 1.573s I1010 20:58:57.825317 2685 leader_election.cc:248] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:57.825479 4167 raft_consensus.cc:1981] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.792s I1010 20:58:57.825495 4167 raft_consensus.cc:1823] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given W1010 20:58:57.830025 2919 consensus_peers.cc:332] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 84fcfff9fdef4acf8c2b47f36330567f. Status: Timed out: UpdateConsensus RPC to 127.6.150.1:46963 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 4 times. W1010 20:58:57.905129 2985 consensus_meta.cc:188] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.657s user 0.000s sys 0.000s I1010 20:58:57.905161 2985 raft_consensus.cc:1677] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42e94fc3adbb419591836d222d946978 in term 3. W1010 20:58:57.905174 2985 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.8:46146 (request call id 4) took 2739ms (client timeout 1614). W1010 20:58:57.905185 2985 rpcz_store.cc:238] Trace: 1010 20:58:55.165637 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:55.165641 (+ 4us) service_pool.cc:202] Handling call 1010 20:58:57.905169 (+2739528us) inbound_call.cc:130] Queueing success response Metrics: {} W1010 20:58:57.905479 3882 consensus_meta.cc:188] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.101s user 0.000s sys 0.000s I1010 20:58:57.905505 3882 raft_consensus.cc:1981] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.602s I1010 20:58:57.905530 3882 raft_consensus.cc:391] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } W1010 20:58:57.905835 3217 consensus_meta.cc:188] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 1.656s user 0.000s sys 0.000s I1010 20:58:57.905886 3217 raft_consensus.cc:1677] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42e94fc3adbb419591836d222d946978 in term 3. W1010 20:58:57.905915 3217 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.8:57074 (request call id 4) took 4271ms (client timeout 1757). W1010 20:58:57.905956 3217 rpcz_store.cc:238] Trace: 1010 20:58:53.634518 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:53.634523 (+ 5us) service_pool.cc:202] Handling call 1010 20:58:57.905909 (+4271386us) inbound_call.cc:130] Queueing success response Metrics: {} I1010 20:58:57.906015 3848 raft_consensus.cc:375] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:57.906033 3848 raft_consensus.cc:2031] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Advancing to term 4 W1010 20:58:57.906406 3755 consensus_meta.cc:188] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 1.658s user 0.000s sys 0.000s I1010 20:58:57.906600 3755 leader_election.cc:223] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 2 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 I1010 20:58:57.906616 3755 leader_election.cc:223] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 2 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 I1010 20:58:57.906641 3337 raft_consensus.cc:1609] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 2: Already voted for candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in this term. I1010 20:58:57.906728 4080 raft_consensus.cc:1981] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.727s I1010 20:58:57.906738 4080 raft_consensus.cc:1823] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Leader election lost for term 1. Reason: None given W1010 20:58:57.906764 3870 consensus_meta.cc:188] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 2.905s user 0.000s sys 0.000s I1010 20:58:57.906781 3870 raft_consensus.cc:1981] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.660s I1010 20:58:57.906803 3870 raft_consensus.cc:391] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } W1010 20:58:57.906816 3889 consensus_meta.cc:188] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 4.334s user 0.000s sys 0.000s I1010 20:58:57.906833 3889 raft_consensus.cc:1981] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.769s I1010 20:58:57.906857 3889 raft_consensus.cc:391] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } W1010 20:58:57.906904 3337 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.1:46701 (request call id 3) took 4272ms (client timeout 1720). W1010 20:58:57.906980 3332 consensus_meta.cc:188] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 1.656s user 0.000s sys 0.000s I1010 20:58:57.907004 3332 raft_consensus.cc:1981] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.995s W1010 20:58:57.907014 3337 rpcz_store.cc:238] Trace: 1010 20:58:53.634328 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:53.634339 (+ 11us) service_pool.cc:202] Handling call 1010 20:58:57.906876 (+4272537us) spinlock_profiling.cc:255] Waited 4.27 s on lock 0x3e5b970. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f3fb2fbf181 00007f3fb1ea747c ffffffffffffffff 1010 20:58:57.906881 (+ 5us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":9826269568} W1010 20:58:57.907090 3863 consensus_meta.cc:188] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 1.659s user 0.000s sys 0.000s I1010 20:58:57.907261 3863 leader_election.cc:223] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 2 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 I1010 20:58:57.907284 3863 leader_election.cc:223] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 2 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 I1010 20:58:57.907322 4102 raft_consensus.cc:1981] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.526s W1010 20:58:57.907367 3794 consensus_meta.cc:188] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 1.657s user 0.000s sys 0.000s I1010 20:58:57.907385 4102 raft_consensus.cc:1823] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Leader election lost for term 1. Reason: None given I1010 20:58:57.907412 2864 raft_consensus.cc:1609] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 in current term 2: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. W1010 20:58:57.907446 2864 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.6:47110 (request call id 4) took 2739ms (client timeout 1591). W1010 20:58:57.907445 2865 consensus_meta.cc:188] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 2.747s user 0.000s sys 0.000s I1010 20:58:57.907467 2865 raft_consensus.cc:1981] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.667s W1010 20:58:57.907469 2864 rpcz_store.cc:238] Trace: 1010 20:58:55.167558 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:55.167597 (+ 39us) service_pool.cc:202] Handling call 1010 20:58:57.907428 (+2739831us) spinlock_profiling.cc:255] Waited 2.74 s on lock 0x3f9e9b0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f9e2c5b9181 00007f9e2b4a147c ffffffffffffffff 1010 20:58:57.907433 (+ 5us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":6301550848} I1010 20:58:57.907551 3794 leader_election.cc:223] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 2 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 W1010 20:58:57.907558 3781 consensus_meta.cc:188] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 2.681s user 0.000s sys 0.000s I1010 20:58:57.907568 3794 leader_election.cc:223] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 2 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 I1010 20:58:57.907577 3781 raft_consensus.cc:1981] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.670s I1010 20:58:57.907601 3781 raft_consensus.cc:391] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } W1010 20:58:57.907618 2631 consensus_meta.cc:188] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 1.659s user 0.000s sys 0.000s I1010 20:58:57.907670 2631 raft_consensus.cc:1677] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de6cfa864d64c58bf4eafb689517c93 in term 2. I1010 20:58:57.907673 3337 raft_consensus.cc:1609] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 in current term 2: Already voted for candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in this term. W1010 20:58:57.907685 2631 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.3:51480 (request call id 3) took 4274ms (client timeout 1574). W1010 20:58:57.907707 2631 rpcz_store.cc:238] Trace: 1010 20:58:53.633392 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:53.633418 (+ 26us) service_pool.cc:202] Handling call 1010 20:58:57.907680 (+4274262us) inbound_call.cc:130] Queueing success response Metrics: {} I1010 20:58:57.907781 2450 leader_election.cc:361] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 2 election: Vote denied by peer 7f623b00f5ae4b9c8e8940692d5dfd44. Message: Invalid argument: T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 in current term 2: Already voted for candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in this term. W1010 20:58:57.907932 2512 consensus_meta.cc:188] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 7.511s user 0.000s sys 0.000s I1010 20:58:57.907963 2512 raft_consensus.cc:1981] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.738s I1010 20:58:57.908438 2511 raft_consensus.cc:1609] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 2: Already voted for candidate 84bde34331b94c4ea003e48f9317f792 in this term. W1010 20:58:57.908524 3715 consensus_meta.cc:188] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 2.709s user 0.000s sys 0.000s W1010 20:58:57.908529 2511 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.1:38540 (request call id 3) took 4274ms (client timeout 1720). W1010 20:58:57.908553 2511 rpcz_store.cc:238] Trace: 1010 20:58:53.634297 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:53.634306 (+ 9us) service_pool.cc:202] Handling call 1010 20:58:57.908501 (+4274195us) spinlock_profiling.cc:255] Waited 4.27 s on lock 0x444c1d0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007ff4d62cf181 00007ff4d51b747c ffffffffffffffff 1010 20:58:57.908525 (+ 24us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":9830443264} I1010 20:58:57.908556 3715 raft_consensus.cc:1981] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.791s I1010 20:58:57.908581 3715 raft_consensus.cc:391] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } W1010 20:58:57.908604 3667 consensus_meta.cc:188] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 2.744s user 0.000s sys 0.000s I1010 20:58:57.908638 3667 raft_consensus.cc:1981] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.733s I1010 20:58:57.908676 3667 raft_consensus.cc:391] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } W1010 20:58:57.908781 3859 consensus_meta.cc:188] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 1.659s user 0.000s sys 0.000s I1010 20:58:57.908939 3859 leader_election.cc:223] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 3 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:58:57.908953 3859 leader_election.cc:223] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 3 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 W1010 20:58:57.909413 3717 consensus_meta.cc:188] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 1.659s user 0.000s sys 0.000s W1010 20:58:57.909420 3789 consensus_meta.cc:188] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 0.640s user 0.000s sys 0.000s I1010 20:58:57.909437 3789 raft_consensus.cc:1981] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.717s I1010 20:58:57.909464 3789 raft_consensus.cc:391] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:58:57.909503 3717 leader_election.cc:223] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 2 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:58:57.909515 3717 leader_election.cc:223] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 2 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:58:57.909529 2508 raft_consensus.cc:1654] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 for term 2 because replica is already servicing an update from a current leader or another vote. I1010 20:58:57.909543 2624 raft_consensus.cc:857] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Refusing update from remote peer 8de6cfa864d64c58bf4eafb689517c93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch) W1010 20:58:57.909577 2508 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.5:46021 (request call id 4) took 4273ms (client timeout 1703). I1010 20:58:57.909586 2510 raft_consensus.cc:1609] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 2: Already voted for candidate 84bde34331b94c4ea003e48f9317f792 in this term. W1010 20:58:57.909610 2508 rpcz_store.cc:238] Trace: 1010 20:58:53.636316 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:53.636325 (+ 9us) service_pool.cc:202] Handling call 1010 20:58:57.909568 (+4273243us) spinlock_profiling.cc:255] Waited 4.27 s on lock 0x409bc10. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007ff4d62cf181 00007ff4d51b747c ffffffffffffffff 1010 20:58:57.909572 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":9828292992} W1010 20:58:57.909627 2624 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.6.150.3:51480 (request call id 4) took 2240ms (client timeout 1000). W1010 20:58:57.909636 3754 consensus_meta.cc:188] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 1.084s user 0.000s sys 0.000s W1010 20:58:57.909656 2510 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.4:40901 (request call id 3) took 5980ms (client timeout 1746). I1010 20:58:57.909665 3754 raft_consensus.cc:1981] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.868s W1010 20:58:57.909670 2624 rpcz_store.cc:238] Trace: 1010 20:58:55.669401 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:55.669408 (+ 7us) service_pool.cc:202] Handling call 1010 20:58:57.909509 (+2240101us) raft_consensus.cc:1068] Updating replica for 0 ops 1010 20:58:57.909553 (+ 44us) raft_consensus.cc:1270] Filling consensus response to leader. 1010 20:58:57.909607 (+ 54us) spinlock_profiling.cc:255] Waited 2.24 s on lock 0x425f4d8. stack: 00000000019065e1 000000000097a767 00000000007880ea 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f324f2ae181 00007f324e19647c ffffffffffffffff 1010 20:58:57.909615 (+ 8us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":5152207232} W1010 20:58:57.909677 2510 rpcz_store.cc:238] Trace: 1010 20:58:51.929035 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:51.929055 (+ 20us) service_pool.cc:202] Handling call 1010 20:58:57.909634 (+5980579us) spinlock_profiling.cc:255] Waited 5.98 s on lock 0x409bc10. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007ff4d62cf181 00007ff4d51b747c ffffffffffffffff 1010 20:58:57.909638 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":13755194240} I1010 20:58:57.909685 3754 raft_consensus.cc:391] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } I1010 20:58:57.909996 2623 raft_consensus.cc:857] T 4aa163baddb74c4998157964dd55b1f4 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Refusing update from remote peer 8de6cfa864d64c58bf4eafb689517c93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch) I1010 20:58:57.910074 3098 raft_consensus.cc:1609] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 in current term 2: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. I1010 20:58:57.910184 4115 consensus_queue.cc:652] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 [LEADER]: Connected to new peer: Peer: 1031512f5120422e9e22a0afabd78f61, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:58:57.910239 2746 raft_consensus.cc:2031] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Advancing to term 3 I1010 20:58:57.910327 2447 leader_election.cc:361] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 2 election: Vote denied by peer 33ef4317e50042ba95e517992caa6531. Message: Invalid argument: T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 in current term 2: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. I1010 20:58:57.910567 2746 raft_consensus.cc:1981] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.770s I1010 20:58:57.910812 3217 raft_consensus.cc:1609] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 2: Already voted for candidate 1380a27d6b824575a3a5e139bee139f4 in this term. I1010 20:58:57.910820 2510 raft_consensus.cc:1609] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 2: Already voted for candidate 84bde34331b94c4ea003e48f9317f792 in this term. I1010 20:58:57.910879 2799 leader_election.cc:361] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 2 election: Vote denied by peer 1380a27d6b824575a3a5e139bee139f4. Message: Invalid argument: T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 2: Already voted for candidate 1380a27d6b824575a3a5e139bee139f4 in this term. I1010 20:58:57.910881 3273 leader_election.cc:361] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 2 election: Vote denied by peer 84bde34331b94c4ea003e48f9317f792. Message: Invalid argument: T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 2: Already voted for candidate 84bde34331b94c4ea003e48f9317f792 in this term. I1010 20:58:57.910900 3218 raft_consensus.cc:1654] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 for term 2 because replica is already servicing an update from a current leader or another vote. W1010 20:58:57.910929 3218 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.2:40008 (request call id 4) took 4277ms (client timeout 1780). W1010 20:58:57.910943 3218 rpcz_store.cc:238] Trace: 1010 20:58:53.633017 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:53.633025 (+ 8us) service_pool.cc:202] Handling call 1010 20:58:57.910922 (+4277897us) spinlock_profiling.cc:255] Waited 4.28 s on lock 0x3b75eb0. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f8ce19fe181 00007f8ce08e647c ffffffffffffffff 1010 20:58:57.910925 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":9839083776} W1010 20:58:57.911742 3808 consensus_meta.cc:188] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 1.662s user 0.000s sys 0.000s I1010 20:58:57.911839 3808 leader_election.cc:223] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 2 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 I1010 20:58:57.911855 3808 leader_election.cc:223] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 2 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 W1010 20:58:57.912267 3673 consensus_meta.cc:188] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6: Time spent flushing consensus metadata: real 1.033s user 0.000s sys 0.000s I1010 20:58:57.912363 3673 leader_election.cc:223] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 3 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 I1010 20:58:57.912397 3673 leader_election.cc:223] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 3 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 I1010 20:58:57.913460 3848 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.875s I1010 20:58:57.913491 3848 raft_consensus.cc:391] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 4 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } W1010 20:58:57.913486 3891 consensus_meta.cc:188] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 1.107s user 0.000s sys 0.000s I1010 20:58:57.913588 3891 leader_election.cc:223] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 3 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:58:57.913601 3891 leader_election.cc:223] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 3 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 I1010 20:58:57.913648 3452 raft_consensus.cc:1609] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 in current term 3: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. W1010 20:58:57.913710 3871 consensus_meta.cc:188] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 0.976s user 0.000s sys 0.000s I1010 20:58:57.913738 3871 raft_consensus.cc:1981] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.680s I1010 20:58:57.913753 3155 leader_election.cc:361] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 3 election: Vote denied by peer 42e94fc3adbb419591836d222d946978. Message: Invalid argument: T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 in current term 3: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:58:57.913760 3871 raft_consensus.cc:391] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:58:57.913836 3218 raft_consensus.cc:1609] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 3: Already voted for candidate 1380a27d6b824575a3a5e139bee139f4 in this term. I1010 20:58:57.913897 3387 leader_election.cc:361] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 3 election: Vote denied by peer 1380a27d6b824575a3a5e139bee139f4. Message: Invalid argument: T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 3: Already voted for candidate 1380a27d6b824575a3a5e139bee139f4 in this term. W1010 20:58:58.248183 2921 leader_election.cc:275] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8de6cfa864d64c58bf4eafb689517c93: Timed out: RequestConsensusVote RPC to 127.6.150.3:40117 timed out after 1.996s W1010 20:58:58.248210 2921 leader_election.cc:275] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 7f623b00f5ae4b9c8e8940692d5dfd44: Timed out: RequestConsensusVote RPC to 127.6.150.7:45436 timed out after 1.996s I1010 20:58:58.248214 2921 leader_election.cc:248] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I1010 20:58:58.248366 4172 raft_consensus.cc:1981] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.634s I1010 20:58:58.248383 4172 raft_consensus.cc:1823] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given I1010 20:58:58.415724 3678 raft_consensus.cc:375] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:58.415743 3678 raft_consensus.cc:2031] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Advancing to term 3 I1010 20:58:58.416159 3678 raft_consensus.cc:1981] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.759s I1010 20:58:58.416208 3678 raft_consensus.cc:391] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } I1010 20:58:58.430367 3698 raft_consensus.cc:375] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:58.430385 3698 raft_consensus.cc:2031] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Advancing to term 3 I1010 20:58:58.500324 3748 raft_consensus.cc:375] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:58.500341 3748 raft_consensus.cc:2031] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Advancing to term 3 I1010 20:58:58.548454 3894 raft_consensus.cc:375] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:58.548467 3894 raft_consensus.cc:2031] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Advancing to term 3 I1010 20:58:58.591081 3722 raft_consensus.cc:375] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:58.591096 3722 raft_consensus.cc:2031] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Advancing to term 3 I1010 20:58:58.591431 3722 raft_consensus.cc:1981] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.568s I1010 20:58:58.591480 3722 raft_consensus.cc:391] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } I1010 20:58:58.662116 3780 raft_consensus.cc:375] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:58.662132 3780 raft_consensus.cc:2031] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Advancing to term 3 I1010 20:58:58.670651 3761 raft_consensus.cc:375] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:58.670699 3761 raft_consensus.cc:2031] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Advancing to term 4 I1010 20:58:58.681334 3697 raft_consensus.cc:375] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:58.681362 3697 raft_consensus.cc:2031] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Advancing to term 3 I1010 20:58:59.194593 3779 raft_consensus.cc:375] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:59.194615 3779 raft_consensus.cc:2031] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Advancing to term 4 I1010 20:58:59.325793 3717 raft_consensus.cc:375] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:59.325824 3717 raft_consensus.cc:2031] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Advancing to term 3 W1010 20:58:59.331118 2919 consensus_peers.cc:332] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 84fcfff9fdef4acf8c2b47f36330567f. Status: Timed out: UpdateConsensus RPC to 127.6.150.1:46963 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 5 times. I1010 20:58:59.410006 3713 raft_consensus.cc:375] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:59.410022 3713 raft_consensus.cc:2031] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Advancing to term 3 W1010 20:58:59.414794 3271 leader_election.cc:275] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1031512f5120422e9e22a0afabd78f61: Timed out: RequestConsensusVote RPC to 127.6.150.1:46963 timed out after 1.506s I1010 20:58:59.414814 3271 leader_election.cc:248] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:59.414958 4173 raft_consensus.cc:1981] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.566s I1010 20:58:59.414974 4173 raft_consensus.cc:1823] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given W1010 20:58:59.426081 3505 leader_election.cc:275] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1031512f5120422e9e22a0afabd78f61: Timed out: RequestConsensusVote RPC to 127.6.150.1:46963 timed out after 1.513s W1010 20:58:59.426861 3507 leader_election.cc:275] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 7f623b00f5ae4b9c8e8940692d5dfd44: Timed out: RequestConsensusVote RPC to 127.6.150.7:45436 timed out after 1.513s I1010 20:58:59.426874 3507 leader_election.cc:248] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I1010 20:58:59.427006 4174 raft_consensus.cc:1981] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.626s I1010 20:58:59.427019 4174 raft_consensus.cc:1823] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given I1010 20:58:59.459033 3881 raft_consensus.cc:375] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:59.459048 3881 raft_consensus.cc:2031] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Advancing to term 3 W1010 20:58:59.477705 2450 leader_election.cc:275] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b8062d66fd284035ae3bc2567bc11e7c: Timed out: RequestConsensusVote RPC to 127.6.150.4:44055 timed out after 1.567s I1010 20:58:59.477721 2450 leader_election.cc:248] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:59.495265 3859 raft_consensus.cc:375] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:59.495313 3859 raft_consensus.cc:2031] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Advancing to term 4 W1010 20:58:59.555521 3389 leader_election.cc:275] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer f6d15ccd64ec49be8396ed78e2a88377: Timed out: RequestConsensusVote RPC to 127.6.150.2:33554 timed out after 1.641s I1010 20:58:59.555536 3389 leader_election.cc:248] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I1010 20:58:59.555681 4176 raft_consensus.cc:1981] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.881s I1010 20:58:59.555716 4176 raft_consensus.cc:1823] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given W1010 20:58:59.562914 3039 leader_election.cc:275] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 84bde34331b94c4ea003e48f9317f792: Timed out: RequestConsensusVote RPC to 127.6.150.0:40113 timed out after 1.650s W1010 20:58:59.562934 3039 leader_election.cc:275] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8de6cfa864d64c58bf4eafb689517c93: Timed out: RequestConsensusVote RPC to 127.6.150.3:40117 timed out after 1.650s I1010 20:58:59.562937 3039 leader_election.cc:248] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:59.563062 4177 raft_consensus.cc:1981] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.619s I1010 20:58:59.563078 4177 raft_consensus.cc:1823] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given W1010 20:58:59.585247 2448 leader_election.cc:275] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1031512f5120422e9e22a0afabd78f61: Timed out: RequestConsensusVote RPC to 127.6.150.1:46963 timed out after 1.674s I1010 20:58:59.585263 2448 leader_election.cc:248] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:59.585420 4178 raft_consensus.cc:1981] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.672s I1010 20:58:59.585435 4178 raft_consensus.cc:1823] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given W1010 20:58:59.611153 3882 consensus_meta.cc:188] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.706s user 0.000s sys 0.000s I1010 20:58:59.611372 3882 leader_election.cc:223] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 3 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:58:59.611392 3882 leader_election.cc:223] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 3 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 I1010 20:58:59.611397 2985 raft_consensus.cc:1584] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 for earlier term 2. Current term is 3. W1010 20:58:59.611450 2985 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.0:43796 (request call id 4) took 1701ms (client timeout 1567). W1010 20:58:59.611487 2985 rpcz_store.cc:238] Trace: 1010 20:58:57.909998 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:57.910041 (+ 43us) service_pool.cc:202] Handling call 1010 20:58:59.611433 (+1701392us) spinlock_profiling.cc:255] Waited 1.7 s on lock 0x4dbe470. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007fe1aafaf181 00007fe1a9e9747c ffffffffffffffff 1010 20:58:59.611438 (+ 5us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3913081728} W1010 20:58:59.611796 3805 consensus_meta.cc:188] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 2.238s user 0.000s sys 0.000s I1010 20:58:59.611836 3805 raft_consensus.cc:1981] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.836s W1010 20:58:59.611855 3894 consensus_meta.cc:188] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.063s user 0.000s sys 0.000s I1010 20:58:59.611871 3805 raft_consensus.cc:391] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:58:59.611881 3894 raft_consensus.cc:1981] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.779s I1010 20:58:59.611909 3894 raft_consensus.cc:391] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } W1010 20:58:59.612027 3870 consensus_meta.cc:188] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 1.705s user 0.000s sys 0.000s I1010 20:58:59.612149 3870 leader_election.cc:223] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 3 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:58:59.612162 3870 leader_election.cc:223] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 3 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:58:59.612210 3335 raft_consensus.cc:1609] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 3: Already voted for candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in this term. W1010 20:58:59.612272 3335 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.8:39867 (request call id 3) took 4446ms (client timeout 1614). I1010 20:58:59.612278 4112 raft_consensus.cc:1981] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.664s W1010 20:58:59.612287 3335 rpcz_store.cc:238] Trace: 1010 20:58:55.165607 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:55.165615 (+ 8us) service_pool.cc:202] Handling call 1010 20:58:59.612265 (+4446650us) spinlock_profiling.cc:255] Waited 4.45 s on lock 0x42161d0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f3fb2fbf181 00007f3fb1ea747c ffffffffffffffff 1010 20:58:59.612269 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":10227113344} I1010 20:58:59.612300 4112 raft_consensus.cc:1823] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: None given W1010 20:58:59.612344 3829 consensus_meta.cc:188] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 2.489s user 0.000s sys 0.000s I1010 20:58:59.612396 3829 raft_consensus.cc:1981] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.711s I1010 20:58:59.612426 3829 raft_consensus.cc:391] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } I1010 20:58:59.612437 3452 raft_consensus.cc:1609] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 3: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:58:59.612483 2985 raft_consensus.cc:1609] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 3: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:58:59.612565 3272 leader_election.cc:361] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 3 election: Vote denied by peer 42e94fc3adbb419591836d222d946978. Message: Invalid argument: T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 3: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:58:59.612607 3273 leader_election.cc:361] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 3 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c. Message: Invalid argument: T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 3: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:58:59.612632 3273 leader_election.cc:248] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I1010 20:58:59.612661 4112 raft_consensus.cc:1981] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.601s I1010 20:58:59.612679 4112 raft_consensus.cc:1823] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given W1010 20:58:59.612747 3678 consensus_meta.cc:188] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 1.197s user 0.000s sys 0.000s W1010 20:58:59.612758 2865 consensus_meta.cc:188] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 1.705s user 0.000s sys 0.000s I1010 20:58:59.612776 2865 raft_consensus.cc:1677] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 580cbaf7092444dfbc6fb344da88d7b6 in term 2. W1010 20:58:59.612790 2865 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.9:41775 (request call id 2) took 4452ms (client timeout 1564). W1010 20:58:59.612808 2865 rpcz_store.cc:238] Trace: 1010 20:58:55.159942 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:55.159957 (+ 15us) service_pool.cc:202] Handling call 1010 20:58:59.612786 (+4452829us) inbound_call.cc:130] Queueing success response Metrics: {"tcmalloc_contention_cycles":896} I1010 20:58:59.612844 3678 leader_election.cc:223] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 3 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 I1010 20:58:59.612865 3678 leader_election.cc:223] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 3 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 W1010 20:58:59.613087 3817 consensus_meta.cc:188] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 3.364s user 0.000s sys 0.000s I1010 20:58:59.613207 3817 leader_election.cc:223] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 2 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 W1010 20:58:59.613220 3780 consensus_meta.cc:188] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 0.951s user 0.000s sys 0.000s I1010 20:58:59.613225 3817 leader_election.cc:223] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 2 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:58:59.613250 3780 raft_consensus.cc:1981] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.953s I1010 20:58:59.613253 3836 raft_consensus.cc:375] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:59.613257 3836 raft_consensus.cc:2031] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Advancing to term 3 I1010 20:58:59.613281 3780 raft_consensus.cc:391] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } I1010 20:58:59.613363 3713 raft_consensus.cc:1981] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.829s I1010 20:58:59.613414 3713 raft_consensus.cc:391] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } I1010 20:58:59.613513 2861 raft_consensus.cc:1609] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 2: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. W1010 20:58:59.613561 2861 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.7:50821 (request call id 2) took 4446ms (client timeout 1534). W1010 20:58:59.613579 2861 rpcz_store.cc:238] Trace: 1010 20:58:55.167095 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:55.167101 (+ 6us) service_pool.cc:202] Handling call 1010 20:58:59.613555 (+4446454us) spinlock_profiling.cc:255] Waited 4.45 s on lock 0x3f9e470. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f9e2c5b9181 00007f9e2b4a147c ffffffffffffffff 1010 20:58:59.613558 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":10226661120} I1010 20:58:59.613636 2862 raft_consensus.cc:1654] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c for term 3 because replica is already servicing an update from a current leader or another vote. W1010 20:58:59.613672 2862 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.4:46161 (request call id 5) took 3363ms (client timeout 1996). W1010 20:58:59.613698 2862 rpcz_store.cc:238] Trace: 1010 20:58:56.250527 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:56.250546 (+ 19us) service_pool.cc:202] Handling call 1010 20:58:59.613665 (+3363119us) spinlock_profiling.cc:255] Waited 3.36 s on lock 0x3f9e470. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f9e2c5b9181 00007f9e2b4a147c ffffffffffffffff 1010 20:58:59.613668 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7735041280} I1010 20:58:59.613863 2985 raft_consensus.cc:1584] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 for earlier term 2. Current term is 3. W1010 20:58:59.613956 2802 leader_election.cc:333] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 2 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c with higher term. Message: Invalid argument: T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 for earlier term 2. Current term is 3. I1010 20:58:59.613981 2802 leader_election.cc:336] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term W1010 20:58:59.614132 3715 consensus_meta.cc:188] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 1.706s user 0.000s sys 0.000s W1010 20:58:59.614140 3789 consensus_meta.cc:188] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 1.705s user 0.000s sys 0.000s W1010 20:58:59.614194 3853 consensus_meta.cc:188] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 3.366s user 0.000s sys 0.000s I1010 20:58:59.614280 4179 raft_consensus.cc:1981] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.553s I1010 20:58:59.614302 4179 raft_consensus.cc:1823] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c with higher term. Message: Invalid argument: T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 for earlier term 2. Current term is 3. W1010 20:58:59.614300 3667 consensus_meta.cc:188] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 1.706s user 0.000s sys 0.000s I1010 20:58:59.614305 3789 leader_election.cc:223] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 3 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:58:59.614352 3789 leader_election.cc:223] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 3 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 W1010 20:58:59.614481 3867 consensus_meta.cc:188] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 3.366s user 0.000s sys 0.000s I1010 20:58:59.614529 3867 raft_consensus.cc:1981] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.610s I1010 20:58:59.614567 3867 raft_consensus.cc:391] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } I1010 20:58:59.614598 3715 leader_election.cc:223] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 3 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 I1010 20:58:59.614622 3715 leader_election.cc:223] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 3 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 I1010 20:58:59.614763 2514 raft_consensus.cc:2031] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Advancing to term 3 I1010 20:58:59.614786 3333 raft_consensus.cc:2031] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Advancing to term 3 W1010 20:58:59.614794 3754 consensus_meta.cc:188] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 1.705s user 0.000s sys 0.000s I1010 20:58:59.614966 3754 leader_election.cc:223] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 3 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:58:59.614995 3754 leader_election.cc:223] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 3 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 W1010 20:58:59.615056 3748 consensus_meta.cc:188] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 1.115s user 0.000s sys 0.000s I1010 20:58:59.615083 3748 raft_consensus.cc:1981] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.792s W1010 20:58:59.615100 2512 consensus_meta.cc:188] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 1.707s user 0.000s sys 0.000s I1010 20:58:59.615111 2512 raft_consensus.cc:1677] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de6cfa864d64c58bf4eafb689517c93 in term 2. I1010 20:58:59.615110 3748 raft_consensus.cc:391] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } W1010 20:58:59.615124 2512 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.3:38215 (request call id 3) took 9218ms (client timeout 1626). I1010 20:58:59.615128 3102 raft_consensus.cc:1609] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 in current term 3: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. W1010 20:58:59.615146 2512 rpcz_store.cc:238] Trace: 1010 20:58:50.396523 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:50.396548 (+ 25us) service_pool.cc:202] Handling call 1010 20:58:59.615119 (+9218571us) inbound_call.cc:130] Queueing success response Metrics: {} I1010 20:58:59.615190 2514 raft_consensus.cc:1981] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.735s I1010 20:58:59.615192 2447 leader_election.cc:361] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 3 election: Vote denied by peer 33ef4317e50042ba95e517992caa6531. Message: Invalid argument: T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 in current term 3: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. I1010 20:58:59.615346 2622 raft_consensus.cc:1584] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 for earlier term 2. Current term is 3. I1010 20:58:59.615352 3853 leader_election.cc:223] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 2 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:58:59.615365 3853 leader_election.cc:223] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 2 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 W1010 20:58:59.615396 2622 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.7:33193 (request call id 4) took 1708ms (client timeout 1506). I1010 20:58:59.615418 4078 raft_consensus.cc:1981] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.724s I1010 20:58:59.615438 4078 raft_consensus.cc:1823] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Leader election lost for term 1. Reason: None given W1010 20:58:59.615438 2622 rpcz_store.cc:238] Trace: 1010 20:58:57.906821 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:57.906831 (+ 10us) service_pool.cc:202] Handling call 1010 20:58:59.615367 (+1708536us) spinlock_profiling.cc:255] Waited 1.71 s on lock 0x3f85970. stack: 1010 20:58:59.615390 (+ 23us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3927921920} I1010 20:58:59.615458 4132 raft_consensus.cc:1981] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.934s I1010 20:58:59.615474 4132 raft_consensus.cc:1823] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: None given I1010 20:58:59.615483 2623 raft_consensus.cc:1654] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 for term 2 because replica is already servicing an update from a current leader or another vote. I1010 20:58:59.615494 3815 raft_consensus.cc:375] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election W1010 20:58:59.615499 2623 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.0:58503 (request call id 3) took 1705ms (client timeout 1674). I1010 20:58:59.615500 3815 raft_consensus.cc:2031] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 2 FOLLOWER]: Advancing to term 3 W1010 20:58:59.615520 2623 rpcz_store.cc:238] Trace: 1010 20:58:57.910185 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:57.910202 (+ 17us) service_pool.cc:202] Handling call 1010 20:58:59.615493 (+1705291us) spinlock_profiling.cc:255] Waited 1.71 s on lock 0x3f85970. stack: 1010 20:58:59.615495 (+ 2us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3922075392} I1010 20:58:59.615527 2626 raft_consensus.cc:1654] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 for term 2 because replica is already servicing an update from a current leader or another vote. I1010 20:58:59.615559 2631 raft_consensus.cc:1609] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 in current term 3: Already voted for candidate 1031512f5120422e9e22a0afabd78f61 in this term. W1010 20:58:59.615573 2626 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.7:33193 (request call id 3) took 7689ms (client timeout 1749). W1010 20:58:59.615592 2631 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.9:55298 (request call id 3) took 1703ms (client timeout 1513). W1010 20:58:59.615614 2631 rpcz_store.cc:238] Trace: 1010 20:58:57.912489 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:57.912497 (+ 8us) service_pool.cc:202] Handling call 1010 20:58:59.615575 (+1703078us) spinlock_profiling.cc:255] Waited 1.7 s on lock 0x3f856d0. stack: 1010 20:58:59.615589 (+ 14us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3916933376} W1010 20:58:59.615615 2626 rpcz_store.cc:238] Trace: 1010 20:58:51.926241 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:51.926273 (+ 32us) service_pool.cc:202] Handling call 1010 20:58:59.615545 (+7689272us) spinlock_profiling.cc:255] Waited 7.69 s on lock 0x3f856d0. stack: 1010 20:58:59.615569 (+ 24us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":17683652992} I1010 20:58:59.615548 2511 raft_consensus.cc:1609] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 3: Already voted for candidate 84bde34331b94c4ea003e48f9317f792 in this term. W1010 20:58:59.615633 3871 consensus_meta.cc:188] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 1.702s user 0.000s sys 0.000s I1010 20:58:59.615710 3220 raft_consensus.cc:1609] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 2: Already voted for candidate 1380a27d6b824575a3a5e139bee139f4 in this term. I1010 20:58:59.615711 3871 leader_election.cc:223] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 3 election: Requesting vote from peer 580cbaf7092444dfbc6fb344da88d7b6 I1010 20:58:59.615726 3871 leader_election.cc:223] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 3 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:58:59.615778 2861 raft_consensus.cc:1609] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 2: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. I1010 20:58:59.615826 2565 leader_election.cc:361] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 2 election: Vote denied by peer 1380a27d6b824575a3a5e139bee139f4. Message: Invalid argument: T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 2: Already voted for candidate 1380a27d6b824575a3a5e139bee139f4 in this term. I1010 20:58:59.615869 2568 leader_election.cc:361] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 2 election: Vote denied by peer 8de6cfa864d64c58bf4eafb689517c93. Message: Invalid argument: T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 2: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. I1010 20:58:59.615890 2568 leader_election.cc:248] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:59.615895 2624 raft_consensus.cc:1654] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 for term 2 because replica is already servicing an update from a current leader or another vote. W1010 20:58:59.615927 2624 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.3:51480 (request call id 6) took 1706ms (client timeout 1754). I1010 20:58:59.615947 4078 raft_consensus.cc:1981] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.506s I1010 20:58:59.615952 4078 raft_consensus.cc:1823] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given W1010 20:58:59.615958 2624 rpcz_store.cc:238] Trace: 1010 20:58:57.909723 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:57.909763 (+ 40us) service_pool.cc:202] Handling call 1010 20:58:59.615921 (+1706158us) spinlock_profiling.cc:255] Waited 1.71 s on lock 0x43401d0. stack: 1010 20:58:59.615923 (+ 2us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3924029696} I1010 20:58:59.615986 2625 raft_consensus.cc:1584] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 for earlier term 1. Current term is 2. I1010 20:58:59.615998 3039 leader_election.cc:361] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 3 election: Vote denied by peer 84bde34331b94c4ea003e48f9317f792. Message: Invalid argument: T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 3: Already voted for candidate 84bde34331b94c4ea003e48f9317f792 in this term. W1010 20:58:59.616015 2625 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.3:51480 (request call id 2) took 7686ms (client timeout 1569). W1010 20:58:59.616027 2625 rpcz_store.cc:238] Trace: 1010 20:58:51.929384 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:51.929395 (+ 11us) service_pool.cc:202] Handling call 1010 20:58:59.615998 (+7686603us) spinlock_profiling.cc:255] Waited 7.69 s on lock 0x43401d0. stack: 1010 20:58:59.616012 (+ 14us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":17679126400} I1010 20:58:59.616031 2800 leader_election.cc:361] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 2 election: Vote denied by peer 1031512f5120422e9e22a0afabd78f61. Message: Service unavailable: T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 for term 2 because replica is already servicing an update from a current leader or another vote. I1010 20:58:59.616053 2800 leader_election.cc:248] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:58:59.616078 3779 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.029s I1010 20:58:59.616096 3667 leader_election.cc:223] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 3 election: Requesting vote from peer 580cbaf7092444dfbc6fb344da88d7b6 I1010 20:58:59.616106 3667 leader_election.cc:223] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 3 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 I1010 20:58:59.616111 3779 raft_consensus.cc:391] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 4 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } I1010 20:58:59.616268 3569 raft_consensus.cc:1609] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 3: Already voted for candidate 580cbaf7092444dfbc6fb344da88d7b6 in this term. I1010 20:58:59.616026 2629 raft_consensus.cc:1654] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 for term 2 because replica is already servicing an update from a current leader or another vote. W1010 20:58:59.616341 2629 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.6:47001 (request call id 2) took 4449ms (client timeout 1591). I1010 20:58:59.616351 2568 leader_election.cc:361] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 3 election: Vote denied by peer 580cbaf7092444dfbc6fb344da88d7b6. Message: Invalid argument: T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 3: Already voted for candidate 580cbaf7092444dfbc6fb344da88d7b6 in this term. W1010 20:58:59.616376 2629 rpcz_store.cc:238] Trace: 1010 20:58:55.166857 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:55.166927 (+ 70us) service_pool.cc:202] Handling call 1010 20:58:59.616332 (+4449405us) spinlock_profiling.cc:255] Waited 4.45 s on lock 0x43401d0. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f324f2ae181 00007f324e19647c ffffffffffffffff 1010 20:58:59.616336 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":10232902016} I1010 20:58:59.616442 3569 raft_consensus.cc:2031] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Advancing to term 3 I1010 20:58:59.616508 4180 raft_consensus.cc:1981] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.668s I1010 20:58:59.616518 4180 raft_consensus.cc:1823] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given W1010 20:58:59.616919 3722 consensus_meta.cc:188] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6: Time spent flushing consensus metadata: real 1.025s user 0.000s sys 0.000s I1010 20:58:59.616931 3836 raft_consensus.cc:1981] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.922s I1010 20:58:59.616971 3836 raft_consensus.cc:391] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } W1010 20:58:59.617013 3781 consensus_meta.cc:188] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 1.709s user 0.000s sys 0.000s I1010 20:58:59.617074 3722 leader_election.cc:223] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 3 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:58:59.617103 3722 leader_election.cc:223] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 3 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:58:59.617131 3781 leader_election.cc:223] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 3 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:58:59.617157 3781 leader_election.cc:223] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 3 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 I1010 20:58:59.617177 2864 raft_consensus.cc:1584] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 for earlier term 2. Current term is 3. W1010 20:58:59.617203 2864 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.5:39587 (request call id 2) took 1705ms (client timeout 1650). W1010 20:58:59.617218 2864 rpcz_store.cc:238] Trace: 1010 20:58:57.911914 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:57.911926 (+ 12us) service_pool.cc:202] Handling call 1010 20:58:59.617197 (+1705271us) spinlock_profiling.cc:255] Waited 1.71 s on lock 0x3f9e1d0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f9e2c5b9181 00007f9e2b4a147c ffffffffffffffff 1010 20:58:59.617199 (+ 2us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3922054528} I1010 20:58:59.617224 3569 raft_consensus.cc:1981] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.626s I1010 20:58:59.617328 3102 raft_consensus.cc:2031] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Advancing to term 3 W1010 20:58:59.617444 3848 consensus_meta.cc:188] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 1.704s user 0.000s sys 0.000s I1010 20:58:59.617544 3848 leader_election.cc:223] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 4 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:58:59.617583 3848 leader_election.cc:223] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 4 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:58:59.617697 2745 raft_consensus.cc:2031] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Advancing to term 4 W1010 20:58:59.617892 3761 consensus_meta.cc:188] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 0.947s user 0.000s sys 0.000s I1010 20:58:59.617913 3761 raft_consensus.cc:1981] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.048s I1010 20:58:59.617935 3761 raft_consensus.cc:391] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 4 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } W1010 20:58:59.618046 2746 consensus_meta.cc:188] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 1.707s user 0.000s sys 0.000s I1010 20:58:59.618090 3859 raft_consensus.cc:1981] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.612s I1010 20:58:59.618104 2746 raft_consensus.cc:1677] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1380a27d6b824575a3a5e139bee139f4 in term 3. W1010 20:58:59.618121 2746 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.6:49570 (request call id 4) took 1707ms (client timeout 1714). I1010 20:58:59.618127 3859 raft_consensus.cc:391] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 4 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } W1010 20:58:59.618134 2746 rpcz_store.cc:238] Trace: 1010 20:58:57.910193 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:57.910224 (+ 31us) service_pool.cc:202] Handling call 1010 20:58:59.618117 (+1707893us) inbound_call.cc:130] Queueing success response Metrics: {} I1010 20:58:59.618160 2748 raft_consensus.cc:1654] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 for term 3 because replica is already servicing an update from a current leader or another vote. W1010 20:58:59.618206 2748 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.8:42100 (request call id 7) took 1704ms (client timeout 1641). W1010 20:58:59.618224 2748 rpcz_store.cc:238] Trace: 1010 20:58:57.913743 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:57.913753 (+ 10us) service_pool.cc:202] Handling call 1010 20:58:59.618199 (+1704446us) spinlock_profiling.cc:255] Waited 1.7 s on lock 0x3419c10. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f7357684181 00007f735656c47c ffffffffffffffff 1010 20:58:59.618203 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3920113920} W1010 20:58:59.618353 3698 consensus_meta.cc:188] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 1.188s user 0.000s sys 0.000s I1010 20:58:59.618376 3698 raft_consensus.cc:1981] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.546s I1010 20:58:59.618403 3698 raft_consensus.cc:391] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } I1010 20:58:59.618705 3155 leader_election.cc:347] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 3 election: Vote granted by peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:58:59.618719 3155 leader_election.cc:248] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. I1010 20:58:59.766660 3819 raft_consensus.cc:375] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:59.766680 3819 raft_consensus.cc:2031] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Advancing to term 3 I1010 20:58:59.787113 3837 raft_consensus.cc:375] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:59.787127 3837 raft_consensus.cc:2031] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [term 2 FOLLOWER]: Advancing to term 3 I1010 20:58:59.888780 3671 raft_consensus.cc:375] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:58:59.888815 3671 raft_consensus.cc:2031] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Advancing to term 3 I1010 20:59:00.097026 3729 raft_consensus.cc:375] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:00.097055 3729 raft_consensus.cc:2031] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Advancing to term 3 I1010 20:59:00.260479 3742 raft_consensus.cc:375] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:00.260499 3742 raft_consensus.cc:2031] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Advancing to term 4 W1010 20:59:00.830762 2919 consensus_peers.cc:332] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 84fcfff9fdef4acf8c2b47f36330567f. Status: Timed out: UpdateConsensus RPC to 127.6.150.1:46963 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 6 times. W1010 20:59:01.019942 3779 consensus_meta.cc:188] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 1.404s user 0.000s sys 0.000s W1010 20:59:01.019923 2514 consensus_meta.cc:188] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 1.405s user 0.000s sys 0.004s I1010 20:59:01.019981 2514 raft_consensus.cc:1677] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1031512f5120422e9e22a0afabd78f61 in term 3. W1010 20:59:01.019995 2514 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.1:38540 (request call id 4) took 1405ms (client timeout 1791). W1010 20:59:01.019990 3894 consensus_meta.cc:188] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.408s user 0.000s sys 0.000s W1010 20:59:01.020028 2514 rpcz_store.cc:238] Trace: 1010 20:58:59.614743 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:59.614753 (+ 10us) service_pool.cc:202] Handling call 1010 20:59:01.019989 (+1405236us) inbound_call.cc:130] Queueing success response Metrics: {"tcmalloc_contention_cycles":65536} W1010 20:59:01.020275 3805 consensus_meta.cc:188] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.408s user 0.000s sys 0.000s W1010 20:59:01.020370 3881 consensus_meta.cc:188] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 1.561s user 0.000s sys 0.000s I1010 20:59:01.020404 3881 raft_consensus.cc:1981] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.855s I1010 20:59:01.020416 2568 leader_election.cc:347] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 3 election: Vote granted by peer 84bde34331b94c4ea003e48f9317f792 I1010 20:59:01.020437 2568 leader_election.cc:248] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. I1010 20:59:01.020447 3881 raft_consensus.cc:391] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } W1010 20:59:01.020444 3829 consensus_meta.cc:188] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 1.408s user 0.000s sys 0.000s I1010 20:59:01.020473 3779 leader_election.cc:223] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 4 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 I1010 20:59:01.020508 3779 leader_election.cc:223] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 4 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:59:01.020570 3452 raft_consensus.cc:1609] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 in current term 4: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:59:01.020656 3452 rpcz_store.cc:248] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.6:42979 (request call id 5) took 1402ms. Request Metrics: {"spinlock_wait_cycles":3226598144} I1010 20:59:01.020668 3220 raft_consensus.cc:1609] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 4: Already voted for candidate 1380a27d6b824575a3a5e139bee139f4 in this term. I1010 20:59:01.020773 3387 leader_election.cc:361] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 4 election: Vote denied by peer 1380a27d6b824575a3a5e139bee139f4. Message: Invalid argument: T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 4: Already voted for candidate 1380a27d6b824575a3a5e139bee139f4 in this term. I1010 20:59:01.020797 3155 leader_election.cc:361] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 4 election: Vote denied by peer 42e94fc3adbb419591836d222d946978. Message: Invalid argument: T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 in current term 4: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. W1010 20:59:01.020817 3332 consensus_meta.cc:188] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 3.114s user 0.000s sys 0.000s I1010 20:59:01.020841 3332 raft_consensus.cc:1677] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8062d66fd284035ae3bc2567bc11e7c in term 3. W1010 20:59:01.020855 3332 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.4:46735 (request call id 5) took 4770ms (client timeout 1996). I1010 20:59:01.020855 4182 raft_consensus.cc:1981] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.817s I1010 20:59:01.020866 4182 raft_consensus.cc:1861] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Leader election won for term 3 W1010 20:59:01.020867 3332 rpcz_store.cc:238] Trace: 1010 20:58:56.250562 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:56.250572 (+ 10us) service_pool.cc:202] Handling call 1010 20:59:01.020851 (+4770279us) inbound_call.cc:130] Queueing success response Metrics: {"tcmalloc_contention_cycles":3328} W1010 20:59:01.021031 3889 consensus_meta.cc:188] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 3.114s user 0.000s sys 0.000s I1010 20:59:01.021141 4182 raft_consensus.cc:467] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [term 3 LEADER]: Becoming Leader. State: Replica: 1031512f5120422e9e22a0afabd78f61, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:59:01.021082 3829 leader_election.cc:223] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 3 election: Requesting vote from peer 580cbaf7092444dfbc6fb344da88d7b6 I1010 20:59:01.021190 4182 consensus_queue.cc:151] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 3, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } W1010 20:59:01.021186 3569 consensus_meta.cc:188] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6: Time spent flushing consensus metadata: real 1.404s user 0.004s sys 0.000s I1010 20:59:01.021211 3569 raft_consensus.cc:1677] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42e94fc3adbb419591836d222d946978 in term 3. I1010 20:59:01.021220 3829 leader_election.cc:223] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 3 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 W1010 20:59:01.021222 3569 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.8:57555 (request call id 3) took 1404ms (client timeout 1680). W1010 20:59:01.021235 3569 rpcz_store.cc:238] Trace: 1010 20:58:59.616424 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:59.616434 (+ 10us) service_pool.cc:202] Handling call 1010 20:59:01.021218 (+1404784us) inbound_call.cc:130] Queueing success response Metrics: {"tcmalloc_contention_cycles":4864} W1010 20:59:01.021277 3761 consensus_meta.cc:188] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 1.403s user 0.000s sys 0.000s I1010 20:59:01.021332 3390 leader_election.cc:347] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 3 election: Vote granted by peer 580cbaf7092444dfbc6fb344da88d7b6 I1010 20:59:01.021340 3390 leader_election.cc:248] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. W1010 20:59:01.021386 3333 consensus_meta.cc:188] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 1.407s user 0.000s sys 0.000s I1010 20:59:01.021401 3333 raft_consensus.cc:1981] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.785s I1010 20:59:01.021404 3761 leader_election.cc:223] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 4 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 I1010 20:59:01.021433 3761 leader_election.cc:223] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 4 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 W1010 20:59:01.021484 3698 consensus_meta.cc:188] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 1.403s user 0.000s sys 0.000s I1010 20:59:01.021510 2514 raft_consensus.cc:2031] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 3 FOLLOWER]: Advancing to term 4 I1010 20:59:01.021553 3744 raft_consensus.cc:375] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:01.021572 3744 raft_consensus.cc:2031] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Advancing to term 4 I1010 20:59:01.021596 3698 leader_election.cc:223] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 3 election: Requesting vote from peer 580cbaf7092444dfbc6fb344da88d7b6 I1010 20:59:01.021601 3336 raft_consensus.cc:1609] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 2: Already voted for candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in this term. I1010 20:59:01.021613 3698 leader_election.cc:223] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 3 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:59:01.021610 2749 raft_consensus.cc:1609] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 in current term 3: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. W1010 20:59:01.021638 3336 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.5:60329 (request call id 3) took 4770ms (client timeout 1513). W1010 20:59:01.021651 3336 rpcz_store.cc:238] Trace: 1010 20:58:56.251600 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:56.251612 (+ 12us) service_pool.cc:202] Handling call 1010 20:59:01.021630 (+4770018us) spinlock_profiling.cc:255] Waited 4.77 s on lock 0x4216470. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f3fb2fbf181 00007f3fb1ea747c ffffffffffffffff 1010 20:59:01.021634 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":10970923776} W1010 20:59:01.021658 2749 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.9:48803 (request call id 3) took 1404ms (client timeout 1568). I1010 20:59:01.021664 3334 raft_consensus.cc:1654] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 for term 2 because replica is already servicing an update from a current leader or another vote. W1010 20:59:01.021674 2749 rpcz_store.cc:238] Trace: 1010 20:58:59.617174 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:59.617185 (+ 11us) service_pool.cc:202] Handling call 1010 20:59:01.021650 (+1404465us) spinlock_profiling.cc:255] Waited 1.4 s on lock 0x34196d0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f7357684181 00007f735656c47c ffffffffffffffff 1010 20:59:01.021654 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3230135040} W1010 20:59:01.021682 3334 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.2:52606 (request call id 2) took 4769ms (client timeout 1573). W1010 20:59:01.021692 3334 rpcz_store.cc:238] Trace: 1010 20:58:56.251832 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:56.251841 (+ 9us) service_pool.cc:202] Handling call 1010 20:59:01.021679 (+4769838us) spinlock_profiling.cc:255] Waited 4.77 s on lock 0x4216470. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f3fb2fbf181 00007f3fb1ea747c ffffffffffffffff 1010 20:59:01.021681 (+ 2us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":10970572160} I1010 20:59:01.021733 3506 leader_election.cc:361] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 3 election: Vote denied by peer f6d15ccd64ec49be8396ed78e2a88377. Message: Invalid argument: T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 in current term 3: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. W1010 20:59:01.021741 3859 consensus_meta.cc:188] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 1.404s user 0.000s sys 0.000s I1010 20:59:01.021811 3859 leader_election.cc:223] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 4 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:59:01.021826 3859 leader_election.cc:223] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 4 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:59:01.021842 4181 raft_consensus.cc:1981] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.010s I1010 20:59:01.021884 4181 raft_consensus.cc:1839] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 4 FOLLOWER]: Leader election decision for defunct term 3: won W1010 20:59:01.022089 3819 consensus_meta.cc:188] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 1.255s user 0.000s sys 0.000s I1010 20:59:01.022114 3819 raft_consensus.cc:1981] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.774s I1010 20:59:01.022136 3819 raft_consensus.cc:391] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } W1010 20:59:01.022137 2745 consensus_meta.cc:188] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 1.404s user 0.000s sys 0.000s I1010 20:59:01.022346 2745 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.966s W1010 20:59:01.022403 3837 consensus_meta.cc:188] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 1.235s user 0.000s sys 0.000s I1010 20:59:01.022428 3837 raft_consensus.cc:1981] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.819s I1010 20:59:01.021579 4185 raft_consensus.cc:1981] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.617s I1010 20:59:01.022459 3837 raft_consensus.cc:391] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } I1010 20:59:01.022467 4185 raft_consensus.cc:1861] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Leader election won for term 3 I1010 20:59:01.022591 2409 catalog_manager.cc:2018] T 95e7c143b0524a9b9ade2cfcc081e679 reported consensus state change: term changed from 0 to 3, leader changed from to 1031512f5120422e9e22a0afabd78f61 (127.6.150.1). New consensus state: current_term: 3 leader_uuid: "1031512f5120422e9e22a0afabd78f61" config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } } I1010 20:59:01.022598 4185 raft_consensus.cc:467] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [term 3 LEADER]: Becoming Leader. State: Replica: 42e94fc3adbb419591836d222d946978, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:59:01.021106 3337 raft_consensus.cc:1609] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 in current term 3: Already voted for candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in this term. I1010 20:59:01.022629 4185 consensus_queue.cc:151] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 3, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } W1010 20:59:01.022645 3337 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.9:42466 (request call id 10) took 3110ms (client timeout 1513). W1010 20:59:01.022666 3337 rpcz_store.cc:238] Trace: 1010 20:58:57.912452 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:57.912459 (+ 7us) service_pool.cc:202] Handling call 1010 20:59:01.022634 (+3110175us) spinlock_profiling.cc:255] Waited 3.11 s on lock 0x3e5beb0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f3fb2fbf181 00007f3fb1ea747c ffffffffffffffff 1010 20:59:01.022639 (+ 5us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7149829504} I1010 20:59:01.022716 3331 raft_consensus.cc:1654] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 for term 3 because replica is already servicing an update from a current leader or another vote. W1010 20:59:01.022747 3331 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.1:46701 (request call id 5) took 1406ms (client timeout 1733). W1010 20:59:01.022773 3331 rpcz_store.cc:238] Trace: 1010 20:58:59.616224 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:59.616250 (+ 26us) service_pool.cc:202] Handling call 1010 20:59:01.022741 (+1406491us) spinlock_profiling.cc:255] Waited 1.41 s on lock 0x3e5beb0. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f3fb2fbf181 00007f3fb1ea747c ffffffffffffffff 1010 20:59:01.022743 (+ 2us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3234801536} I1010 20:59:01.022841 2568 leader_election.cc:361] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 3 election: Vote denied by peer 7f623b00f5ae4b9c8e8940692d5dfd44. Message: Service unavailable: T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 for term 3 because replica is already servicing an update from a current leader or another vote. I1010 20:59:01.022855 2568 leader_election.cc:248] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I1010 20:59:01.022930 3218 raft_consensus.cc:1584] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 for earlier term 3. Current term is 4. I1010 20:59:01.022960 3217 raft_consensus.cc:1654] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 for term 3 because replica is already servicing an update from a current leader or another vote. I1010 20:59:01.022963 3569 raft_consensus.cc:1609] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 3: Already voted for candidate 580cbaf7092444dfbc6fb344da88d7b6 in this term. W1010 20:59:01.022972 3218 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.5:44141 (request call id 3) took 1409ms (client timeout 1759). W1010 20:59:01.022987 3218 rpcz_store.cc:238] Trace: 1010 20:58:59.613555 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:59.613563 (+ 8us) service_pool.cc:202] Handling call 1010 20:59:01.022965 (+1409402us) spinlock_profiling.cc:255] Waited 1.41 s on lock 0x3b75430. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f8ce19fe181 00007f8ce08e647c ffffffffffffffff 1010 20:59:01.022968 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3241494784} W1010 20:59:01.022991 3217 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.0:60940 (request call id 3) took 1407ms (client timeout 1868). W1010 20:59:01.023005 3217 rpcz_store.cc:238] Trace: 1010 20:58:59.615070 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:59.615089 (+ 19us) service_pool.cc:202] Handling call 1010 20:59:01.022984 (+1407895us) spinlock_profiling.cc:255] Waited 1.41 s on lock 0x3b75430. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f8ce19fe181 00007f8ce08e647c ffffffffffffffff 1010 20:59:01.022987 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3238073728} W1010 20:59:01.023066 3036 leader_election.cc:333] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 3 election: Vote denied by peer 1380a27d6b824575a3a5e139bee139f4 with higher term. Message: Invalid argument: T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 for earlier term 3. Current term is 4. I1010 20:59:01.023077 3036 leader_election.cc:336] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term W1010 20:59:01.023092 2447 leader_election.cc:333] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 3 election: Vote denied by peer 1380a27d6b824575a3a5e139bee139f4 with higher term. Message: Service unavailable: T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 for term 3 because replica is already servicing an update from a current leader or another vote. I1010 20:59:01.023100 2447 leader_election.cc:336] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term I1010 20:59:01.023156 4191 raft_consensus.cc:1981] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.604s I1010 20:59:01.023177 2749 raft_consensus.cc:2031] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Advancing to term 4 I1010 20:59:01.023226 4191 raft_consensus.cc:1823] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given I1010 20:59:01.023250 3452 raft_consensus.cc:2031] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Advancing to term 4 I1010 20:59:01.023258 3273 leader_election.cc:361] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 3 election: Vote denied by peer 580cbaf7092444dfbc6fb344da88d7b6. Message: Invalid argument: T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 3: Already voted for candidate 580cbaf7092444dfbc6fb344da88d7b6 in this term. I1010 20:59:01.023344 2625 raft_consensus.cc:1609] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 3: Already voted for candidate 1031512f5120422e9e22a0afabd78f61 in this term. I1010 20:59:01.023432 2403 catalog_manager.cc:2018] T 7836528275cb45aab827361611bb3bbe reported consensus state change: term changed from 0 to 3, leader changed from to 42e94fc3adbb419591836d222d946978 (127.6.150.8). New consensus state: current_term: 3 leader_uuid: "42e94fc3adbb419591836d222d946978" config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } } I1010 20:59:01.023444 3101 raft_consensus.cc:2031] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Advancing to term 4 I1010 20:59:01.023465 3271 leader_election.cc:361] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 3 election: Vote denied by peer 1031512f5120422e9e22a0afabd78f61. Message: Invalid argument: T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 3: Already voted for candidate 1031512f5120422e9e22a0afabd78f61 in this term. I1010 20:59:01.023475 3271 leader_election.cc:248] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I1010 20:59:01.021162 3889 leader_election.cc:223] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 2 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:59:01.023567 3889 leader_election.cc:223] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 2 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:59:01.023701 3099 raft_consensus.cc:1609] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 2: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. I1010 20:59:01.023711 2746 raft_consensus.cc:1609] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 2: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:59:01.023808 3270 leader_election.cc:361] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 2 election: Vote denied by peer 33ef4317e50042ba95e517992caa6531. Message: Invalid argument: T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 2: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. I1010 20:59:01.023835 3272 leader_election.cc:361] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 2 election: Vote denied by peer f6d15ccd64ec49be8396ed78e2a88377. Message: Invalid argument: T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 2: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:59:01.023854 3272 leader_election.cc:248] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. W1010 20:59:01.023910 3748 consensus_meta.cc:188] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 1.409s user 0.000s sys 0.000s I1010 20:59:01.023977 4197 raft_consensus.cc:1981] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.744s I1010 20:59:01.023994 4197 raft_consensus.cc:1823] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given I1010 20:59:01.024036 3748 leader_election.cc:223] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 3 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:59:01.024054 3748 leader_election.cc:223] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 3 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 W1010 20:59:01.024137 3815 consensus_meta.cc:188] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 1.409s user 0.000s sys 0.000s I1010 20:59:01.024175 3815 raft_consensus.cc:1981] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.713s I1010 20:59:01.024220 3815 raft_consensus.cc:391] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } W1010 20:59:01.024551 3780 consensus_meta.cc:188] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 1.411s user 0.000s sys 0.000s W1010 20:59:01.024615 3836 consensus_meta.cc:188] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 1.408s user 0.000s sys 0.000s W1010 20:59:01.024668 3713 consensus_meta.cc:188] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 1.411s user 0.000s sys 0.000s I1010 20:59:01.024731 3780 leader_election.cc:223] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 3 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:59:01.024752 3780 leader_election.cc:223] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 3 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 I1010 20:59:01.024770 3098 raft_consensus.cc:1609] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 3: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. W1010 20:59:01.024798 3098 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.4:56288 (request call id 3) took 1413ms (client timeout 1602). I1010 20:59:01.024822 3836 leader_election.cc:223] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 3 election: Requesting vote from peer 580cbaf7092444dfbc6fb344da88d7b6 W1010 20:59:01.024826 3098 rpcz_store.cc:238] Trace: 1010 20:58:59.611461 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:59.611498 (+ 37us) service_pool.cc:202] Handling call 1010 20:59:01.024792 (+1413294us) spinlock_profiling.cc:255] Waited 1.41 s on lock 0x4d75c10. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007fc8aa54b181 00007fc8a943347c ffffffffffffffff 1010 20:59:01.024795 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3250498816} I1010 20:59:01.024837 3836 leader_election.cc:223] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 3 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:59:01.024837 3713 leader_election.cc:223] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 3 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:59:01.024852 3713 leader_election.cc:223] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 3 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 I1010 20:59:01.024899 3099 raft_consensus.cc:1609] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 3: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. I1010 20:59:01.024898 2918 leader_election.cc:361] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 3 election: Vote denied by peer 33ef4317e50042ba95e517992caa6531. Message: Invalid argument: T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 3: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. W1010 20:59:01.024953 3867 consensus_meta.cc:188] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 1.410s user 0.000s sys 0.000s W1010 20:59:01.024976 3102 consensus_meta.cc:188] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 1.408s user 0.000s sys 0.000s I1010 20:59:01.024997 3102 raft_consensus.cc:1981] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.618s I1010 20:59:01.025038 2983 raft_consensus.cc:1609] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 3: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:59:01.025038 2565 leader_election.cc:361] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 3 election: Vote denied by peer 33ef4317e50042ba95e517992caa6531. Message: Invalid argument: T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 3: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. I1010 20:59:01.025087 2625 raft_consensus.cc:1609] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 3: Already voted for candidate 1031512f5120422e9e22a0afabd78f61 in this term. I1010 20:59:01.025097 3867 leader_election.cc:223] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 3 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:59:01.025123 3867 leader_election.cc:223] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 3 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:59:01.025122 3039 leader_election.cc:361] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 3 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c. Message: Invalid argument: T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 3: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:59:01.025141 3037 leader_election.cc:361] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 3 election: Vote denied by peer 1031512f5120422e9e22a0afabd78f61. Message: Invalid argument: T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 3: Already voted for candidate 1031512f5120422e9e22a0afabd78f61 in this term. W1010 20:59:01.025491 3671 consensus_meta.cc:188] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 1.137s user 0.000s sys 0.000s I1010 20:59:01.025512 3671 raft_consensus.cc:1981] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.541s I1010 20:59:01.025540 3671 raft_consensus.cc:391] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } W1010 20:59:01.025596 3697 consensus_meta.cc:188] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 2.344s user 0.000s sys 0.000s I1010 20:59:01.025636 3697 raft_consensus.cc:1981] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.860s I1010 20:59:01.025662 3697 raft_consensus.cc:391] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:59:01.025825 2861 raft_consensus.cc:1609] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 3: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. I1010 20:59:01.025871 3453 raft_consensus.cc:1640] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [term 3 LEADER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 for term 3 because replica is either leader or believes a valid leader to be alive. W1010 20:59:01.025881 2861 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.8:60818 (request call id 5) took 1409ms (client timeout 1680). W1010 20:59:01.025894 2861 rpcz_store.cc:238] Trace: 1010 20:58:59.616393 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:59.616423 (+ 30us) service_pool.cc:202] Handling call 1010 20:59:01.025872 (+1409449us) spinlock_profiling.cc:255] Waited 1.41 s on lock 0x3f9e710. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f9e2c5b9181 00007f9e2b4a147c ffffffffffffffff 1010 20:59:01.025876 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3241567104} I1010 20:59:01.025928 2627 raft_consensus.cc:825] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Rejecting Update request from peer b8062d66fd284035ae3bc2567bc11e7c for earlier term 2. Current term is 3. Ops: [] I1010 20:59:01.025974 2801 leader_election.cc:361] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 3 election: Vote denied by peer 42e94fc3adbb419591836d222d946978. Message: Invalid argument: T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [term 3 LEADER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 for term 3 because replica is either leader or believes a valid leader to be alive. W1010 20:59:01.025992 2627 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.6.150.4:41994 (request call id 3) took 8591ms (client timeout 1000). I1010 20:59:01.025996 2630 raft_consensus.cc:825] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Rejecting Update request from peer b8062d66fd284035ae3bc2567bc11e7c for earlier term 2. Current term is 3. Ops: [] W1010 20:59:01.026026 2627 rpcz_store.cc:238] Trace: 1010 20:58:52.434736 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:52.434746 (+ 10us) service_pool.cc:202] Handling call 1010 20:58:56.248840 (+3814094us) raft_consensus.cc:1068] Updating replica for 0 ops 1010 20:59:01.025945 (+4777105us) raft_consensus.cc:1270] Filling consensus response to leader. 1010 20:59:01.025970 (+ 25us) spinlock_profiling.cc:255] Waited 4.78 s on lock 0x4340470. stack: 00000000019065e1 0000000000978c4a 000000000097a5b3 00000000007880ea 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f324f2ae181 00007f324e19647c ffffffffffffffff 1010 20:59:01.025984 (+ 14us) spinlock_profiling.cc:255] Waited 3.81 s on lock 0x43a2458. stack: 00000000019065e1 000000000097a767 00000000007880ea 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f324f2ae181 00007f324e19647c ffffffffffffffff 1010 20:59:01.025988 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":19759651456} W1010 20:59:01.026062 2630 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.6.150.4:41994 (request call id 6) took 4196ms (client timeout 1000). W1010 20:59:01.026084 2630 rpcz_store.cc:238] Trace: 1010 20:58:56.829753 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:56.829772 (+ 19us) service_pool.cc:202] Handling call 1010 20:59:01.025986 (+4196214us) raft_consensus.cc:1068] Updating replica for 0 ops 1010 20:59:01.026017 (+ 31us) raft_consensus.cc:1270] Filling consensus response to leader. 1010 20:59:01.026042 (+ 25us) spinlock_profiling.cc:255] Waited 4.2 s on lock 0x43a2458. stack: 00000000019065e1 000000000097a767 00000000007880ea 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f324f2ae181 00007f324e19647c ffffffffffffffff 1010 20:59:01.026045 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":9651260288} I1010 20:59:01.026083 2621 raft_consensus.cc:825] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Rejecting Update request from peer b8062d66fd284035ae3bc2567bc11e7c for earlier term 2. Current term is 3. Ops: [] I1010 20:59:01.026118 2629 raft_consensus.cc:825] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Rejecting Update request from peer b8062d66fd284035ae3bc2567bc11e7c for earlier term 2. Current term is 3. Ops: [] W1010 20:59:01.026124 2621 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.6.150.4:41994 (request call id 7) took 2696ms (client timeout 1000). W1010 20:59:01.026149 2621 rpcz_store.cc:238] Trace: 1010 20:58:58.329933 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:58.329958 (+ 25us) service_pool.cc:202] Handling call 1010 20:59:01.026030 (+2696072us) raft_consensus.cc:1068] Updating replica for 0 ops 1010 20:59:01.026096 (+ 66us) raft_consensus.cc:1270] Filling consensus response to leader. 1010 20:59:01.026115 (+ 19us) spinlock_profiling.cc:255] Waited 2.7 s on lock 0x43a2458. stack: 00000000019065e1 000000000097a767 00000000007880ea 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f324f2ae181 00007f324e19647c ffffffffffffffff 1010 20:59:01.026119 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":6200920960,"tcmalloc_contention_cycles":768} W1010 20:59:01.026154 2629 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.6.150.4:41994 (request call id 8) took 1195ms (client timeout 1000). W1010 20:59:01.026170 2629 rpcz_store.cc:238] Trace: 1010 20:58:59.830236 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:59.830257 (+ 21us) service_pool.cc:202] Handling call 1010 20:59:01.026109 (+1195852us) raft_consensus.cc:1068] Updating replica for 0 ops 1010 20:59:01.026127 (+ 18us) raft_consensus.cc:1270] Filling consensus response to leader. 1010 20:59:01.026146 (+ 19us) spinlock_profiling.cc:255] Waited 1.2 s on lock 0x43a2458. stack: 00000000019065e1 000000000097a767 00000000007880ea 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f324f2ae181 00007f324e19647c ffffffffffffffff 1010 20:59:01.026149 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":2750432512} I1010 20:59:01.026187 2632 raft_consensus.cc:825] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Rejecting Update request from peer b8062d66fd284035ae3bc2567bc11e7c for earlier term 2. Current term is 3. Ops: [] I1010 20:59:01.026208 2628 raft_consensus.cc:825] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Rejecting Update request from peer b8062d66fd284035ae3bc2567bc11e7c for earlier term 2. Current term is 3. Ops: [] W1010 20:59:01.026212 2632 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.6.150.4:41994 (request call id 4) took 7091ms (client timeout 1000). W1010 20:59:01.026224 2632 rpcz_store.cc:238] Trace: 1010 20:58:53.935065 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:53.935096 (+ 31us) service_pool.cc:202] Handling call 1010 20:59:01.026181 (+7091085us) raft_consensus.cc:1068] Updating replica for 0 ops 1010 20:59:01.026192 (+ 11us) raft_consensus.cc:1270] Filling consensus response to leader. 1010 20:59:01.026208 (+ 16us) spinlock_profiling.cc:255] Waited 7.09 s on lock 0x43a2458. stack: 00000000019065e1 000000000097a767 00000000007880ea 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f324f2ae181 00007f324e19647c ffffffffffffffff 1010 20:59:01.026210 (+ 2us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":16309468416} W1010 20:59:01.026232 2628 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.6.150.4:41994 (request call id 5) took 5696ms (client timeout 1000). W1010 20:59:01.026244 2628 rpcz_store.cc:238] Trace: 1010 20:58:55.329398 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:55.329445 (+ 47us) service_pool.cc:202] Handling call 1010 20:59:01.026202 (+5696757us) raft_consensus.cc:1068] Updating replica for 0 ops 1010 20:59:01.026214 (+ 12us) raft_consensus.cc:1270] Filling consensus response to leader. 1010 20:59:01.026226 (+ 12us) spinlock_profiling.cc:255] Waited 5.7 s on lock 0x43a2458. stack: 00000000019065e1 000000000097a767 00000000007880ea 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f324f2ae181 00007f324e19647c ffffffffffffffff 1010 20:59:01.026229 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":13102511488} I1010 20:59:01.026404 3805 leader_election.cc:223] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 3 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 I1010 20:59:01.026412 3453 raft_consensus.cc:1640] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 for term 3 because replica is either leader or believes a valid leader to be alive. I1010 20:59:01.026414 2985 raft_consensus.cc:1609] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 in current term 3: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:59:01.026424 3805 leader_election.cc:223] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 3 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:59:01.026430 3894 leader_election.cc:223] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 3 election: Requesting vote from peer 580cbaf7092444dfbc6fb344da88d7b6 I1010 20:59:01.026443 3894 leader_election.cc:223] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 3 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 W1010 20:59:01.026451 2985 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.6:46028 (request call id 3) took 1411ms (client timeout 1717). W1010 20:59:01.026465 2985 rpcz_store.cc:238] Trace: 1010 20:58:59.614458 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:59.614481 (+ 23us) service_pool.cc:202] Handling call 1010 20:59:01.026445 (+1411964us) spinlock_profiling.cc:255] Waited 1.41 s on lock 0x4a3beb0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007fe1aafaf181 00007fe1a9e9747c ffffffffffffffff 1010 20:59:01.026447 (+ 2us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3247411072} I1010 20:59:01.026484 2567 leader_election.cc:361] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 3 election: Vote denied by peer 42e94fc3adbb419591836d222d946978. Message: Invalid argument: T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 for term 3 because replica is either leader or believes a valid leader to be alive. I1010 20:59:01.026504 3217 raft_consensus.cc:1609] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 3: Already voted for candidate 1380a27d6b824575a3a5e139bee139f4 in this term. I1010 20:59:01.026515 2744 raft_consensus.cc:1609] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 3: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:59:01.026523 3156 leader_election.cc:361] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 3 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c. Message: Invalid argument: T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 in current term 3: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:59:01.026562 2920 leader_election.cc:361] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 3 election: Vote denied by peer f6d15ccd64ec49be8396ed78e2a88377. Message: Invalid argument: T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 3: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:59:01.026564 2986 raft_consensus.cc:1609] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 in current term 3: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:59:01.026568 3569 raft_consensus.cc:1609] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 3: Already voted for candidate 580cbaf7092444dfbc6fb344da88d7b6 in this term. W1010 20:59:01.026602 2986 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.9:33842 (request call id 4) took 1409ms (client timeout 1568). W1010 20:59:01.026618 2986 rpcz_store.cc:238] Trace: 1010 20:58:59.617266 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:59.617295 (+ 29us) service_pool.cc:202] Handling call 1010 20:59:01.026596 (+1409301us) spinlock_profiling.cc:255] Waited 1.41 s on lock 0x4dbe710. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007fe1aafaf181 00007fe1a9e9747c ffffffffffffffff 1010 20:59:01.026598 (+ 2us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3241291520} I1010 20:59:01.026624 2685 leader_election.cc:361] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 3 election: Vote denied by peer 580cbaf7092444dfbc6fb344da88d7b6. Message: Invalid argument: T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 3: Already voted for candidate 580cbaf7092444dfbc6fb344da88d7b6 in this term. I1010 20:59:01.026633 2984 raft_consensus.cc:1654] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 for term 3 because replica is already servicing an update from a current leader or another vote. I1010 20:59:01.026690 3390 leader_election.cc:361] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 3 election: Vote denied by peer 8de6cfa864d64c58bf4eafb689517c93. Message: Invalid argument: T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 3: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. I1010 20:59:01.026706 3569 raft_consensus.cc:1609] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 3: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:59:01.026718 2685 leader_election.cc:361] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 3 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c. Message: Service unavailable: T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 for term 3 because replica is already servicing an update from a current leader or another vote. I1010 20:59:01.026727 2685 leader_election.cc:248] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I1010 20:59:01.026760 2918 leader_election.cc:361] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 3 election: Vote denied by peer 1380a27d6b824575a3a5e139bee139f4. Message: Invalid argument: T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 3: Already voted for candidate 1380a27d6b824575a3a5e139bee139f4 in this term. I1010 20:59:01.026769 2802 leader_election.cc:361] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 3 election: Vote denied by peer 580cbaf7092444dfbc6fb344da88d7b6. Message: Invalid argument: T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 3: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:59:01.026777 2802 leader_election.cc:248] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I1010 20:59:01.026818 3507 leader_election.cc:361] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 3 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c. Message: Invalid argument: T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 in current term 3: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:59:01.026826 3507 leader_election.cc:248] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I1010 20:59:01.026856 2984 raft_consensus.cc:1640] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 2 LEADER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 for term 3 because replica is either leader or believes a valid leader to be alive. I1010 20:59:01.026893 3569 raft_consensus.cc:1609] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 3: Already voted for candidate 580cbaf7092444dfbc6fb344da88d7b6 in this term. I1010 20:59:01.026916 4200 raft_consensus.cc:1981] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.721s I1010 20:59:01.026926 4200 raft_consensus.cc:1823] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given I1010 20:59:01.026939 4198 raft_consensus.cc:1981] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.564s I1010 20:59:01.026953 4198 raft_consensus.cc:1823] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given I1010 20:59:01.026978 2921 leader_election.cc:361] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 3 election: Vote denied by peer 580cbaf7092444dfbc6fb344da88d7b6. Message: Invalid argument: T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 3: Already voted for candidate 580cbaf7092444dfbc6fb344da88d7b6 in this term. I1010 20:59:01.026988 2921 leader_election.cc:248] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I1010 20:59:01.027025 4199 raft_consensus.cc:1981] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.770s I1010 20:59:01.027040 4199 raft_consensus.cc:1823] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given I1010 20:59:01.027075 4196 raft_consensus.cc:1981] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.662s I1010 20:59:01.027088 4196 raft_consensus.cc:1823] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given I1010 20:59:01.027124 4201 raft_consensus.cc:1981] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.510s I1010 20:59:01.027139 4201 raft_consensus.cc:1823] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given I1010 20:59:01.027153 2568 leader_election.cc:361] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 3 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c. Message: Invalid argument: T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 2 LEADER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 for term 3 because replica is either leader or believes a valid leader to be alive. I1010 20:59:01.027163 2568 leader_election.cc:248] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I1010 20:59:01.027289 4202 raft_consensus.cc:1981] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.541s I1010 20:59:01.027302 4202 raft_consensus.cc:1823] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given I1010 20:59:01.193279 3720 raft_consensus.cc:375] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:01.193306 3720 raft_consensus.cc:2031] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Advancing to term 4 I1010 20:59:01.193567 3720 raft_consensus.cc:1981] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.503s I1010 20:59:01.193603 3720 raft_consensus.cc:391] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } W1010 20:59:01.214915 2921 leader_election.cc:275] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 84bde34331b94c4ea003e48f9317f792: Timed out: RequestConsensusVote RPC to 127.6.150.0:40113 timed out after 1.602s I1010 20:59:01.214941 2921 leader_election.cc:248] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I1010 20:59:01.215129 4203 raft_consensus.cc:1981] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.891s I1010 20:59:01.215170 4203 raft_consensus.cc:1823] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given I1010 20:59:01.267645 3834 raft_consensus.cc:375] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:01.267663 3834 raft_consensus.cc:2031] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 2 FOLLOWER]: Advancing to term 3 W1010 20:59:01.287434 2802 leader_election.cc:275] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7f623b00f5ae4b9c8e8940692d5dfd44: Timed out: RequestConsensusVote RPC to 127.6.150.7:45436 timed out after 1.672s W1010 20:59:01.288445 2799 leader_election.cc:275] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 33ef4317e50042ba95e517992caa6531: Timed out: RequestConsensusVote RPC to 127.6.150.5:40764 timed out after 1.670s W1010 20:59:01.288553 2802 leader_election.cc:275] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 84bde34331b94c4ea003e48f9317f792: Timed out: RequestConsensusVote RPC to 127.6.150.0:40113 timed out after 1.670s I1010 20:59:01.288563 2802 leader_election.cc:248] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I1010 20:59:01.288692 4204 raft_consensus.cc:1981] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.562s I1010 20:59:01.288708 4204 raft_consensus.cc:1823] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given I1010 20:59:01.330602 2628 raft_consensus.cc:825] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Rejecting Update request from peer b8062d66fd284035ae3bc2567bc11e7c for earlier term 2. Current term is 3. Ops: [] I1010 20:59:01.330709 4092 consensus_queue.cc:662] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [LEADER]: Peer responded invalid term: Peer: 1031512f5120422e9e22a0afabd78f61, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:59:01.330889 4205 raft_consensus.cc:2026] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 2 LEADER]: Stepping down as leader of term 2 I1010 20:59:01.330914 4205 raft_consensus.cc:502] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b8062d66fd284035ae3bc2567bc11e7c, State: 1, Role: LEADER Watermarks: {Received: term: 2 index: 1 Committed: term: 2 index: 1} I1010 20:59:01.330926 4205 consensus_queue.cc:168] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Current term: 2, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:59:01.330996 4205 consensus_peers.cc:357] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c -> Peer 42e94fc3adbb419591836d222d946978 (127.6.150.8:35299): Closing peer: 42e94fc3adbb419591836d222d946978 I1010 20:59:01.331035 4205 consensus_peers.cc:357] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Closing peer: 1031512f5120422e9e22a0afabd78f61 I1010 20:59:01.331053 4205 raft_consensus.cc:2031] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 2 FOLLOWER]: Advancing to term 3 W1010 20:59:01.332468 3156 leader_election.cc:275] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8de6cfa864d64c58bf4eafb689517c93: Timed out: RequestConsensusVote RPC to 127.6.150.3:40117 timed out after 1.717s I1010 20:59:01.332504 3156 leader_election.cc:248] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I1010 20:59:01.332671 4206 raft_consensus.cc:1981] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.597s I1010 20:59:01.332695 4206 raft_consensus.cc:1823] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given W1010 20:59:01.407069 2568 leader_election.cc:275] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 7f623b00f5ae4b9c8e8940692d5dfd44: Timed out: RequestConsensusVote RPC to 127.6.150.7:45436 timed out after 1.791s W1010 20:59:01.494570 3155 leader_election.cc:275] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer f6d15ccd64ec49be8396ed78e2a88377: Timed out: RequestConsensusVote RPC to 127.6.150.2:33554 timed out after 1.875s I1010 20:59:01.494590 3155 leader_election.cc:248] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. I1010 20:59:01.494743 4207 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.862s I1010 20:59:01.494758 4207 raft_consensus.cc:1823] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: None given I1010 20:59:01.521935 2507 raft_consensus.cc:857] T 95e7c143b0524a9b9ade2cfcc081e679 P 84bde34331b94c4ea003e48f9317f792 [term 3 FOLLOWER]: Refusing update from remote peer 1031512f5120422e9e22a0afabd78f61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 1. (index mismatch) I1010 20:59:01.522095 4182 consensus_queue.cc:652] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [LEADER]: Connected to new peer: Peer: 84bde34331b94c4ea003e48f9317f792, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:59:01.522930 3569 raft_consensus.cc:857] T 7836528275cb45aab827361611bb3bbe P 580cbaf7092444dfbc6fb344da88d7b6 [term 3 FOLLOWER]: Refusing update from remote peer 42e94fc3adbb419591836d222d946978: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 1. (index mismatch) I1010 20:59:01.523075 4185 consensus_queue.cc:652] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [LEADER]: Connected to new peer: Peer: 580cbaf7092444dfbc6fb344da88d7b6, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:59:01.523164 2864 raft_consensus.cc:857] T 7836528275cb45aab827361611bb3bbe P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Refusing update from remote peer 42e94fc3adbb419591836d222d946978: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 1. (index mismatch) I1010 20:59:01.523284 4185 consensus_queue.cc:652] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 [LEADER]: Connected to new peer: Peer: 8de6cfa864d64c58bf4eafb689517c93, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:59:01.659548 3869 raft_consensus.cc:375] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:01.659561 3869 raft_consensus.cc:2031] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Advancing to term 4 I1010 20:59:01.804568 3842 raft_consensus.cc:375] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:01.804594 3842 raft_consensus.cc:2031] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 2 FOLLOWER]: Advancing to term 3 W1010 20:59:02.522697 2568 consensus_peers.cc:332] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 -> Peer 7f623b00f5ae4b9c8e8940692d5dfd44 (127.6.150.7:45436): Couldn't send request to peer 7f623b00f5ae4b9c8e8940692d5dfd44 for tablet 95e7c143b0524a9b9ade2cfcc081e679. Status: Timed out: UpdateConsensus RPC to 127.6.150.7:45436 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 1 times. W1010 20:59:02.560611 3881 consensus_meta.cc:188] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 1.540s user 0.000s sys 0.000s W1010 20:59:02.560631 3815 consensus_meta.cc:188] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 1.536s user 0.000s sys 0.000s I1010 20:59:02.560808 3881 leader_election.cc:223] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 3 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 W1010 20:59:02.560811 3717 consensus_meta.cc:188] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 3.235s user 0.000s sys 0.000s I1010 20:59:02.560835 3881 leader_election.cc:223] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 3 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:59:02.560861 3717 raft_consensus.cc:1981] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.836s I1010 20:59:02.560891 3717 raft_consensus.cc:391] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } I1010 20:59:02.560956 3815 leader_election.cc:223] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 3 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:59:02.560992 3815 leader_election.cc:223] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 3 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:59:02.561038 2510 raft_consensus.cc:1654] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 for term 2 because replica is already servicing an update from a current leader or another vote. W1010 20:59:02.561020 3742 consensus_meta.cc:188] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 2.301s user 0.000s sys 0.000s I1010 20:59:02.561059 3742 raft_consensus.cc:1981] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.976s W1010 20:59:02.561084 2510 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.5:46021 (request call id 6) took 4649ms (client timeout 1650). I1010 20:59:02.561086 3742 raft_consensus.cc:391] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 4 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } W1010 20:59:02.561120 2510 rpcz_store.cc:238] Trace: 1010 20:58:57.911951 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:57.911976 (+ 25us) service_pool.cc:202] Handling call 1010 20:59:02.561060 (+4649084us) spinlock_profiling.cc:255] Waited 4.65 s on lock 0x444c470. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007ff4d62cf181 00007ff4d51b747c ffffffffffffffff 1010 20:59:02.561078 (+ 18us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":10692762368} I1010 20:59:02.561179 2513 raft_consensus.cc:1654] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 for term 1 because replica is already servicing an update from a current leader or another vote. I1010 20:59:02.561208 2511 raft_consensus.cc:1609] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 3: Already voted for candidate 84bde34331b94c4ea003e48f9317f792 in this term. W1010 20:59:02.561223 2513 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.5:46021 (request call id 3) took 10634ms (client timeout 1553). W1010 20:59:02.561238 2513 rpcz_store.cc:238] Trace: 1010 20:58:51.926965 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:51.926975 (+ 10us) service_pool.cc:202] Handling call 1010 20:59:02.561216 (+10634241us) spinlock_profiling.cc:255] Waited 10.6 s on lock 0x444c470. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007ff4d62cf181 00007ff4d51b747c ffffffffffffffff 1010 20:59:02.561219 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":24458543360} W1010 20:59:02.561265 2511 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.3:38215 (request call id 15) took 2943ms (client timeout 1670). W1010 20:59:02.561281 2511 rpcz_store.cc:238] Trace: 1010 20:58:59.617464 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:59.617473 (+ 9us) service_pool.cc:202] Handling call 1010 20:59:02.561245 (+2943772us) spinlock_profiling.cc:255] Waited 2.94 s on lock 0x444c470. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007ff4d62cf181 00007ff4d51b747c ffffffffffffffff 1010 20:59:02.561250 (+ 5us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":6770555648} I1010 20:59:02.561300 4067 raft_consensus.cc:1981] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.616s I1010 20:59:02.561323 4067 raft_consensus.cc:1823] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 3 FOLLOWER]: Leader election lost for term 1. Reason: Vote denied by peer 8de6cfa864d64c58bf4eafb689517c93 with higher term. Message: Invalid argument: T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 for earlier term 1. Current term is 2. I1010 20:59:02.561410 2865 raft_consensus.cc:1609] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 in current term 3: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. I1010 20:59:02.561534 2450 leader_election.cc:361] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 3 election: Vote denied by peer 8de6cfa864d64c58bf4eafb689517c93. Message: Invalid argument: T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 in current term 3: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. W1010 20:59:02.562552 3333 consensus_meta.cc:188] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 1.541s user 0.000s sys 0.000s I1010 20:59:02.562579 3333 raft_consensus.cc:1677] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1031512f5120422e9e22a0afabd78f61 in term 3. W1010 20:59:02.562611 3333 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.1:46701 (request call id 4) took 2947ms (client timeout 1791). W1010 20:59:02.562621 3333 rpcz_store.cc:238] Trace: 1010 20:58:59.614757 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:59.614778 (+ 21us) service_pool.cc:202] Handling call 1010 20:59:02.562607 (+2947829us) inbound_call.cc:130] Queueing success response Metrics: {} I1010 20:59:02.562665 3331 raft_consensus.cc:857] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Refusing update from remote peer 1031512f5120422e9e22a0afabd78f61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 1. (index mismatch) W1010 20:59:02.562719 3331 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.6.150.1:46701 (request call id 6) took 1041ms (client timeout 1000). W1010 20:59:02.562732 3331 rpcz_store.cc:238] Trace: 1010 20:59:01.521710 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:59:01.521732 (+ 22us) service_pool.cc:202] Handling call 1010 20:59:02.562650 (+1040918us) raft_consensus.cc:1068] Updating replica for 0 ops 1010 20:59:02.562695 (+ 45us) raft_consensus.cc:1270] Filling consensus response to leader. 1010 20:59:02.562713 (+ 18us) spinlock_profiling.cc:255] Waited 1.04 s on lock 0x4100a58. stack: 00000000019065e1 000000000097a767 00000000007880ea 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f3fb2fbf181 00007f3fb1ea747c ffffffffffffffff 1010 20:59:02.562716 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":2394093952} W1010 20:59:02.562938 3744 consensus_meta.cc:188] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 1.541s user 0.000s sys 0.000s I1010 20:59:02.563050 3744 raft_consensus.cc:1981] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.094s I1010 20:59:02.563089 3744 raft_consensus.cc:391] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 4 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } W1010 20:59:02.563657 3819 consensus_meta.cc:188] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 1.541s user 0.000s sys 0.000s I1010 20:59:02.563810 3819 leader_election.cc:223] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 3 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:59:02.563824 3819 leader_election.cc:223] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 3 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 I1010 20:59:02.563827 2747 raft_consensus.cc:1654] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 for term 3 because replica is already servicing an update from a current leader or another vote. W1010 20:59:02.563865 2747 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.5:40398 (request call id 5) took 1538ms (client timeout 1829). W1010 20:59:02.563881 2747 rpcz_store.cc:238] Trace: 1010 20:59:01.024951 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:59:01.024959 (+ 8us) service_pool.cc:202] Handling call 1010 20:59:02.563858 (+1538899us) spinlock_profiling.cc:255] Waited 1.54 s on lock 0x37cc1d0. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f7357684181 00007f735656c47c ffffffffffffffff 1010 20:59:02.563861 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3539361024} I1010 20:59:02.563941 2746 raft_consensus.cc:1609] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 3: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:59:02.563946 3098 raft_consensus.cc:1609] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 3: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. W1010 20:59:02.563971 2746 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.1:40603 (request call id 3) took 1539ms (client timeout 1792). W1010 20:59:02.563987 2746 rpcz_store.cc:238] Trace: 1010 20:59:01.024131 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:59:01.024150 (+ 19us) service_pool.cc:202] Handling call 1010 20:59:02.563966 (+1539816us) spinlock_profiling.cc:255] Waited 1.54 s on lock 0x37cc1d0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f7357684181 00007f735656c47c ffffffffffffffff 1010 20:59:02.563968 (+ 2us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3541439872} W1010 20:59:02.564076 3729 consensus_meta.cc:188] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 2.467s user 0.000s sys 0.000s I1010 20:59:02.564105 3729 raft_consensus.cc:1981] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.786s I1010 20:59:02.564108 2628 raft_consensus.cc:1609] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 3: Already voted for candidate 1031512f5120422e9e22a0afabd78f61 in this term. I1010 20:59:02.564127 2567 leader_election.cc:361] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 3 election: Vote denied by peer f6d15ccd64ec49be8396ed78e2a88377. Message: Invalid argument: T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 3: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:59:02.564138 2567 leader_election.cc:248] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. W1010 20:59:02.564173 2745 consensus_meta.cc:188] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 1.542s user 0.000s sys 0.000s I1010 20:59:02.564187 2745 raft_consensus.cc:1677] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1380a27d6b824575a3a5e139bee139f4 in term 4. I1010 20:59:02.564187 2683 leader_election.cc:361] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 3 election: Vote denied by peer 1031512f5120422e9e22a0afabd78f61. Message: Invalid argument: T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 3: Already voted for candidate 1031512f5120422e9e22a0afabd78f61 in this term. W1010 20:59:02.564205 2745 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.6:49570 (request call id 5) took 2946ms (client timeout 1875). I1010 20:59:02.564230 2682 leader_election.cc:361] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 3 election: Vote denied by peer 33ef4317e50042ba95e517992caa6531. Message: Invalid argument: T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 3: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. W1010 20:59:02.564237 2745 rpcz_store.cc:238] Trace: 1010 20:58:59.617672 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:59.617681 (+ 9us) service_pool.cc:202] Handling call 1010 20:59:02.564201 (+2946520us) inbound_call.cc:130] Queueing success response Metrics: {} I1010 20:59:02.564242 2682 leader_election.cc:248] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I1010 20:59:02.564229 3729 raft_consensus.cc:391] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } I1010 20:59:02.564301 2748 raft_consensus.cc:1654] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 for term 4 because replica is already servicing an update from a current leader or another vote. I1010 20:59:02.564401 4213 raft_consensus.cc:1981] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.905s I1010 20:59:02.564410 4213 raft_consensus.cc:1823] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given I1010 20:59:02.564415 3763 raft_consensus.cc:375] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:02.564422 3763 raft_consensus.cc:2031] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Advancing to term 5 W1010 20:59:02.564437 2748 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.8:42100 (request call id 8) took 1543ms (client timeout 2029). W1010 20:59:02.564450 2748 rpcz_store.cc:238] Trace: 1010 20:59:01.020969 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:59:01.020999 (+ 30us) service_pool.cc:202] Handling call 1010 20:59:02.564421 (+1543422us) spinlock_profiling.cc:255] Waited 1.54 s on lock 0x3419eb0. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f7357684181 00007f735656c47c ffffffffffffffff 1010 20:59:02.564424 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3549565312} I1010 20:59:02.564518 3389 leader_election.cc:361] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 4 election: Vote denied by peer f6d15ccd64ec49be8396ed78e2a88377. Message: Service unavailable: T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 for term 4 because replica is already servicing an update from a current leader or another vote. I1010 20:59:02.564530 3389 leader_election.cc:248] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. I1010 20:59:02.564615 3038 leader_election.cc:361] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 3 election: Vote denied by peer f6d15ccd64ec49be8396ed78e2a88377. Message: Service unavailable: T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 for term 3 because replica is already servicing an update from a current leader or another vote. I1010 20:59:02.564630 3038 leader_election.cc:248] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I1010 20:59:02.564736 4215 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.077s I1010 20:59:02.564780 4215 raft_consensus.cc:1823] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: None given I1010 20:59:02.564791 4216 raft_consensus.cc:1981] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.818s I1010 20:59:02.564812 4214 raft_consensus.cc:1981] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.949s I1010 20:59:02.564826 4216 raft_consensus.cc:1823] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given I1010 20:59:02.564833 4214 raft_consensus.cc:1823] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given W1010 20:59:02.564857 3720 consensus_meta.cc:188] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.371s user 0.000s sys 0.000s I1010 20:59:02.565026 3720 leader_election.cc:223] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 4 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:59:02.565042 3720 leader_election.cc:223] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 4 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 W1010 20:59:02.565121 3869 consensus_meta.cc:188] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 0.906s user 0.000s sys 0.000s I1010 20:59:02.565140 3869 raft_consensus.cc:1981] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.780s I1010 20:59:02.565160 3869 raft_consensus.cc:391] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:59:02.565188 3331 raft_consensus.cc:2031] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Advancing to term 4 W1010 20:59:02.565206 4205 consensus_meta.cc:188] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.234s user 0.000s sys 0.000s W1010 20:59:02.565855 3837 consensus_meta.cc:188] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 1.543s user 0.000s sys 0.000s I1010 20:59:02.565961 3837 leader_election.cc:223] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 3 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:59:02.566017 3837 leader_election.cc:223] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 3 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 I1010 20:59:02.566092 2865 raft_consensus.cc:2031] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Advancing to term 3 I1010 20:59:02.566155 2628 raft_consensus.cc:2031] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [term 2 FOLLOWER]: Advancing to term 3 I1010 20:59:02.566375 2628 raft_consensus.cc:1981] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.904s W1010 20:59:02.568778 3102 consensus_meta.cc:188] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 1.544s user 0.000s sys 0.000s I1010 20:59:02.568797 3102 raft_consensus.cc:1677] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de6cfa864d64c58bf4eafb689517c93 in term 3. W1010 20:59:02.568811 3102 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.3:53121 (request call id 3) took 2951ms (client timeout 1670). W1010 20:59:02.568821 3102 rpcz_store.cc:238] Trace: 1010 20:58:59.617304 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:59.617319 (+ 15us) service_pool.cc:202] Handling call 1010 20:59:02.568807 (+2951488us) inbound_call.cc:130] Queueing success response Metrics: {} I1010 20:59:02.568842 3099 raft_consensus.cc:1654] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 for term 3 because replica is already servicing an update from a current leader or another vote. I1010 20:59:02.568960 2447 leader_election.cc:361] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 3 election: Vote denied by peer 33ef4317e50042ba95e517992caa6531. Message: Service unavailable: T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 for term 3 because replica is already servicing an update from a current leader or another vote. I1010 20:59:02.568974 2447 leader_election.cc:248] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I1010 20:59:02.569005 4067 raft_consensus.cc:1981] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.852s I1010 20:59:02.569015 4067 raft_consensus.cc:1823] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given W1010 20:59:02.569015 3101 consensus_meta.cc:188] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 1.546s user 0.000s sys 0.000s I1010 20:59:02.569033 3101 raft_consensus.cc:1981] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.649s W1010 20:59:02.569936 3671 consensus_meta.cc:188] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 1.544s user 0.000s sys 0.000s I1010 20:59:02.570070 3671 leader_election.cc:223] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 3 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:59:02.570099 3671 leader_election.cc:223] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 3 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:59:02.570101 2864 raft_consensus.cc:1609] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 3: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. I1010 20:59:02.570219 3451 raft_consensus.cc:1609] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 3: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:59:02.570225 3390 leader_election.cc:361] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 3 election: Vote denied by peer 8de6cfa864d64c58bf4eafb689517c93. Message: Invalid argument: T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 3: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. W1010 20:59:02.570250 3697 consensus_meta.cc:188] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 1.545s user 0.000s sys 0.000s I1010 20:59:02.570363 2801 leader_election.cc:361] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 3 election: Vote denied by peer 42e94fc3adbb419591836d222d946978. Message: Invalid argument: T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 3: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:59:02.570371 3697 leader_election.cc:223] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 3 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 I1010 20:59:02.570385 3697 leader_election.cc:223] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 3 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:59:02.570390 2861 raft_consensus.cc:1654] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c for term 3 because replica is already servicing an update from a current leader or another vote. W1010 20:59:02.570436 2861 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.4:46161 (request call id 6) took 1543ms (client timeout 1836). I1010 20:59:02.570436 2862 raft_consensus.cc:1609] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 in current term 3: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. W1010 20:59:02.570475 2861 rpcz_store.cc:238] Trace: 1010 20:59:01.027034 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:59:01.027044 (+ 10us) service_pool.cc:202] Handling call 1010 20:59:02.570423 (+1543379us) spinlock_profiling.cc:255] Waited 1.54 s on lock 0x3c2dc10. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f9e2c5b9181 00007f9e2b4a147c ffffffffffffffff 1010 20:59:02.570431 (+ 8us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3549667584} W1010 20:59:02.570510 2862 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.6:47110 (request call id 5) took 2956ms (client timeout 1717). W1010 20:59:02.570539 2862 rpcz_store.cc:238] Trace: 1010 20:58:59.614426 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:59.614433 (+ 7us) service_pool.cc:202] Handling call 1010 20:59:02.570491 (+2956058us) spinlock_profiling.cc:255] Waited 2.96 s on lock 0x3c2dc10. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f9e2c5b9181 00007f9e2b4a147c ffffffffffffffff 1010 20:59:02.570507 (+ 16us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":6798716288} I1010 20:59:02.570668 2865 raft_consensus.cc:1981] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.958s I1010 20:59:02.570922 3217 raft_consensus.cc:1609] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 3: Already voted for candidate 1380a27d6b824575a3a5e139bee139f4 in this term. I1010 20:59:02.570947 2921 leader_election.cc:361] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 3 election: Vote denied by peer 8de6cfa864d64c58bf4eafb689517c93. Message: Service unavailable: T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c for term 3 because replica is already servicing an update from a current leader or another vote. I1010 20:59:02.570956 2921 leader_election.cc:248] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I1010 20:59:02.570955 2984 raft_consensus.cc:1609] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 3: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:59:02.570997 2799 leader_election.cc:361] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 3 election: Vote denied by peer 1380a27d6b824575a3a5e139bee139f4. Message: Invalid argument: T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 3: Already voted for candidate 1380a27d6b824575a3a5e139bee139f4 in this term. I1010 20:59:02.571105 4217 raft_consensus.cc:1981] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.879s I1010 20:59:02.571118 4217 raft_consensus.cc:1823] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given I1010 20:59:02.571144 2802 leader_election.cc:361] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 3 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c. Message: Invalid argument: T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 3: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:59:02.571156 2802 leader_election.cc:248] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I1010 20:59:02.571274 4218 raft_consensus.cc:1981] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.942s I1010 20:59:02.571285 4218 raft_consensus.cc:1823] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given W1010 20:59:02.635653 3155 leader_election.cc:275] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 42e94fc3adbb419591836d222d946978: Timed out: RequestConsensusVote RPC to 127.6.150.8:35299 timed out after 1.612s W1010 20:59:02.635696 3155 leader_election.cc:275] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer f6d15ccd64ec49be8396ed78e2a88377: Timed out: RequestConsensusVote RPC to 127.6.150.2:33554 timed out after 1.612s I1010 20:59:02.635700 3155 leader_election.cc:248] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. I1010 20:59:02.635846 4219 raft_consensus.cc:1981] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.102s I1010 20:59:02.635867 4219 raft_consensus.cc:1823] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: None given I1010 20:59:02.794406 3673 raft_consensus.cc:375] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 3 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:02.794447 3673 raft_consensus.cc:2031] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 3 FOLLOWER]: Advancing to term 4 I1010 20:59:02.794713 3673 raft_consensus.cc:1981] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.634s I1010 20:59:02.794741 3673 raft_consensus.cc:391] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 4 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } W1010 20:59:02.979213 3039 leader_election.cc:275] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 84bde34331b94c4ea003e48f9317f792: Timed out: RequestConsensusVote RPC to 127.6.150.0:40113 timed out after 1.953s I1010 20:59:02.979233 3039 leader_election.cc:248] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I1010 20:59:02.979375 4220 raft_consensus.cc:1981] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.691s I1010 20:59:02.979395 4220 raft_consensus.cc:1823] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given I1010 20:59:02.998986 3816 raft_consensus.cc:371] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Failure of leader b8062d66fd284035ae3bc2567bc11e7c detected. Triggering leader election I1010 20:59:02.999002 3816 raft_consensus.cc:2031] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 2 FOLLOWER]: Advancing to term 3 I1010 20:59:03.014235 3764 raft_consensus.cc:375] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:03.014251 3764 raft_consensus.cc:2031] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Advancing to term 4 I1010 20:59:03.022110 3333 raft_consensus.cc:857] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Refusing update from remote peer 1031512f5120422e9e22a0afabd78f61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 1. (index mismatch) I1010 20:59:03.022289 4182 consensus_queue.cc:652] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 [LEADER]: Connected to new peer: Peer: 7f623b00f5ae4b9c8e8940692d5dfd44, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false W1010 20:59:03.071353 3156 leader_election.cc:275] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 84bde34331b94c4ea003e48f9317f792: Timed out: RequestConsensusVote RPC to 127.6.150.0:40113 timed out after 2.048s W1010 20:59:03.073210 3153 leader_election.cc:275] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 33ef4317e50042ba95e517992caa6531: Timed out: RequestConsensusVote RPC to 127.6.150.5:40764 timed out after 2.048s I1010 20:59:03.073235 3153 leader_election.cc:248] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. I1010 20:59:03.073438 4225 raft_consensus.cc:1981] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.824s I1010 20:59:03.073453 4225 raft_consensus.cc:1823] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: None given I1010 20:59:03.146168 3817 raft_consensus.cc:375] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:03.146188 3817 raft_consensus.cc:2031] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 2 FOLLOWER]: Advancing to term 3 W1010 20:59:04.068614 2920 leader_election.cc:275] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 42e94fc3adbb419591836d222d946978: Timed out: RequestConsensusVote RPC to 127.6.150.8:35299 timed out after 1.503s W1010 20:59:04.068768 2921 leader_election.cc:275] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 7f623b00f5ae4b9c8e8940692d5dfd44: Timed out: RequestConsensusVote RPC to 127.6.150.7:45436 timed out after 1.503s I1010 20:59:04.068781 2921 leader_election.cc:248] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. I1010 20:59:04.068948 4226 raft_consensus.cc:1981] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.817s I1010 20:59:04.068964 4226 raft_consensus.cc:1823] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: None given W1010 20:59:04.112087 2801 leader_election.cc:275] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer f6d15ccd64ec49be8396ed78e2a88377: Timed out: RequestConsensusVote RPC to 127.6.150.2:33554 timed out after 1.541s I1010 20:59:04.112128 2801 leader_election.cc:248] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I1010 20:59:04.112347 4227 raft_consensus.cc:1981] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.982s I1010 20:59:04.112365 4227 raft_consensus.cc:1823] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given I1010 20:59:04.297195 3722 raft_consensus.cc:375] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 3 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:04.297235 3722 raft_consensus.cc:2031] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 3 FOLLOWER]: Advancing to term 4 W1010 20:59:04.308627 3673 consensus_meta.cc:188] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6: Time spent flushing consensus metadata: real 1.514s user 0.000s sys 0.000s W1010 20:59:04.308645 3717 consensus_meta.cc:188] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 1.748s user 0.000s sys 0.000s W1010 20:59:04.308652 3742 consensus_meta.cc:188] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 1.748s user 0.000s sys 0.000s W1010 20:59:04.309132 2514 consensus_meta.cc:188] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 3.288s user 0.000s sys 0.000s I1010 20:59:04.309150 2514 raft_consensus.cc:1981] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.176s W1010 20:59:04.309178 3452 consensus_meta.cc:188] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 3.286s user 0.000s sys 0.000s I1010 20:59:04.309195 3452 raft_consensus.cc:1981] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.582s I1010 20:59:04.309281 3722 raft_consensus.cc:1981] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.125s I1010 20:59:04.309326 3722 raft_consensus.cc:391] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 4 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } W1010 20:59:04.309424 3869 consensus_meta.cc:188] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.744s user 0.000s sys 0.000s W1010 20:59:04.309662 3729 consensus_meta.cc:188] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 1.745s user 0.000s sys 0.000s W1010 20:59:04.309788 2749 consensus_meta.cc:188] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 3.287s user 0.000s sys 0.000s I1010 20:59:04.309844 2749 raft_consensus.cc:1981] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.074s I1010 20:59:04.309937 3717 leader_election.cc:223] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 3 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:59:04.310016 3717 leader_election.cc:223] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 3 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 W1010 20:59:04.310025 2865 consensus_meta.cc:188] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 1.739s user 0.000s sys 0.000s I1010 20:59:04.310044 2865 raft_consensus.cc:1677] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1380a27d6b824575a3a5e139bee139f4 in term 3. I1010 20:59:04.310051 4175 raft_consensus.cc:1981] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.841s W1010 20:59:04.310065 2865 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.6:47110 (request call id 6) took 1743ms (client timeout 1819). W1010 20:59:04.310080 2865 rpcz_store.cc:238] Trace: 1010 20:59:02.566075 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:59:02.566083 (+ 8us) service_pool.cc:202] Handling call 1010 20:59:04.310059 (+1743976us) inbound_call.cc:130] Queueing success response Metrics: {} I1010 20:59:04.310084 3742 leader_election.cc:223] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 4 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:59:04.310087 4175 raft_consensus.cc:1823] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: None given I1010 20:59:04.310088 3729 leader_election.cc:223] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 3 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:59:04.310104 3742 leader_election.cc:223] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 4 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 I1010 20:59:04.310106 3729 leader_election.cc:223] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 3 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:59:04.310106 2512 raft_consensus.cc:1654] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 for term 3 because replica is already servicing an update from a current leader or another vote. I1010 20:59:04.310123 3453 raft_consensus.cc:1609] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 4: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. W1010 20:59:04.310153 2512 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.5:46021 (request call id 8) took 3285ms (client timeout 1953). W1010 20:59:04.310169 3453 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.4:56561 (request call id 25) took 1745ms (client timeout 1503). I1010 20:59:04.310178 2748 raft_consensus.cc:1654] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 for term 3 because replica is already servicing an update from a current leader or another vote. W1010 20:59:04.310185 2512 rpcz_store.cc:238] Trace: 1010 20:59:01.025001 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:59:01.025092 (+ 91us) service_pool.cc:202] Handling call 1010 20:59:04.310129 (+3285037us) spinlock_profiling.cc:255] Waited 3.29 s on lock 0x409bc10. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007ff4d62cf181 00007ff4d51b747c ffffffffffffffff 1010 20:59:04.310134 (+ 5us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7555511168} W1010 20:59:04.310189 3453 rpcz_store.cc:238] Trace: 1010 20:59:02.565109 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:59:02.565145 (+ 36us) service_pool.cc:202] Handling call 1010 20:59:04.310161 (+1745016us) spinlock_profiling.cc:255] Waited 1.74 s on lock 0x3e4f6d0. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007fd760afa181 00007fd75f9e247c ffffffffffffffff 1010 20:59:04.310165 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":4013424000} W1010 20:59:04.310250 2748 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.3:58643 (request call id 3) took 1740ms (client timeout 1541). I1010 20:59:04.310251 2744 raft_consensus.cc:1609] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 3: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. W1010 20:59:04.310268 2748 rpcz_store.cc:238] Trace: 1010 20:59:02.570199 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:59:02.570204 (+ 5us) service_pool.cc:202] Handling call 1010 20:59:04.310241 (+1740037us) spinlock_profiling.cc:255] Waited 1.74 s on lock 0x3419970. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f7357684181 00007f735656c47c ffffffffffffffff 1010 20:59:04.310245 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":4001899264} W1010 20:59:04.310307 2744 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.8:42100 (request call id 9) took 1749ms (client timeout 1855). W1010 20:59:04.310346 2744 rpcz_store.cc:238] Trace: 1010 20:59:02.560915 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:59:02.560924 (+ 9us) service_pool.cc:202] Handling call 1010 20:59:04.310298 (+1749374us) spinlock_profiling.cc:255] Waited 1.75 s on lock 0x3419970. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f7357684181 00007f735656c47c ffffffffffffffff 1010 20:59:04.310302 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":4023411968} I1010 20:59:04.310375 2865 raft_consensus.cc:1609] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 3: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. I1010 20:59:04.310390 3156 leader_election.cc:347] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 3 election: Vote granted by peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:59:04.310428 3156 leader_election.cc:248] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. I1010 20:59:04.310431 3389 leader_election.cc:361] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 3 election: Vote denied by peer f6d15ccd64ec49be8396ed78e2a88377. Message: Invalid argument: T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 3: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:59:04.310453 3389 leader_election.cc:248] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I1010 20:59:04.310530 3869 leader_election.cc:223] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 4 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 I1010 20:59:04.310560 2685 leader_election.cc:361] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 3 election: Vote denied by peer 8de6cfa864d64c58bf4eafb689517c93. Message: Invalid argument: T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 3: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. I1010 20:59:04.310564 3869 leader_election.cc:223] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 4 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:59:04.310602 3673 leader_election.cc:223] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 4 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 I1010 20:59:04.310626 3673 leader_election.cc:223] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 4 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 I1010 20:59:04.310708 2984 raft_consensus.cc:1609] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 4: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. W1010 20:59:04.310706 3817 consensus_meta.cc:188] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 1.165s user 0.000s sys 0.000s I1010 20:59:04.310750 3817 raft_consensus.cc:1981] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.645s I1010 20:59:04.310788 3817 raft_consensus.cc:391] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:59:04.310793 2632 raft_consensus.cc:2031] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Advancing to term 4 I1010 20:59:04.310860 3390 leader_election.cc:361] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 4 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c. Message: Invalid argument: T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 4: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:59:04.310966 4228 raft_consensus.cc:1981] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.535s I1010 20:59:04.310977 4228 raft_consensus.cc:1861] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Leader election won for term 3 I1010 20:59:04.311085 3453 raft_consensus.cc:1609] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 3: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:59:04.311111 4228 raft_consensus.cc:467] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [term 3 LEADER]: Becoming Leader. State: Replica: 1380a27d6b824575a3a5e139bee139f4, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:59:04.311131 4229 raft_consensus.cc:1981] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.553s I1010 20:59:04.311141 4229 raft_consensus.cc:1823] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given I1010 20:59:04.311144 2508 raft_consensus.cc:1609] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 3: Already voted for candidate 84bde34331b94c4ea003e48f9317f792 in this term. I1010 20:59:04.311156 4228 consensus_queue.cc:151] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 3, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } W1010 20:59:04.311202 2508 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.4:40901 (request call id 13) took 4699ms (client timeout 1602). W1010 20:59:04.311228 2508 rpcz_store.cc:238] Trace: 1010 20:58:59.611689 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:59.611723 (+ 34us) service_pool.cc:202] Handling call 1010 20:59:04.311194 (+4699471us) spinlock_profiling.cc:255] Waited 4.7 s on lock 0x409bc10. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007ff4d62cf181 00007ff4d51b747c ffffffffffffffff 1010 20:59:04.311198 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":10808633216} I1010 20:59:04.311348 2684 leader_election.cc:361] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 3 election: Vote denied by peer 42e94fc3adbb419591836d222d946978. Message: Invalid argument: T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 3: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:59:04.311370 2684 leader_election.cc:248] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I1010 20:59:04.311494 2984 raft_consensus.cc:1609] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 in current term 3: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:59:04.311542 4232 raft_consensus.cc:1981] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.578s I1010 20:59:04.311552 4232 raft_consensus.cc:1823] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given I1010 20:59:04.311583 2450 leader_election.cc:361] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 3 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c. Message: Invalid argument: T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 in current term 3: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:59:04.311640 3099 raft_consensus.cc:1609] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 in current term 3: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. I1010 20:59:04.311754 3334 raft_consensus.cc:2031] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Advancing to term 4 I1010 20:59:04.311856 2447 leader_election.cc:361] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 3 election: Vote denied by peer 33ef4317e50042ba95e517992caa6531. Message: Invalid argument: T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 in current term 3: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. I1010 20:59:04.311864 2447 leader_election.cc:248] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I1010 20:59:04.311933 4175 raft_consensus.cc:1981] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.940s I1010 20:59:04.311945 4175 raft_consensus.cc:1823] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given I1010 20:59:04.313563 2409 catalog_manager.cc:2018] T 981834f5e88f4585b6420fca7ddcb289 reported consensus state change: term changed from 0 to 3, leader changed from to 1380a27d6b824575a3a5e139bee139f4 (127.6.150.6). New consensus state: current_term: 3 leader_uuid: "1380a27d6b824575a3a5e139bee139f4" config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } } W1010 20:59:04.314749 3744 consensus_meta.cc:188] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 1.752s user 0.004s sys 0.000s I1010 20:59:04.314944 4146 raft_consensus.cc:1981] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.911s I1010 20:59:04.314955 3744 leader_election.cc:223] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 4 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:59:04.314980 3744 leader_election.cc:223] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 4 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:59:04.315009 4146 raft_consensus.cc:1823] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 4 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c with higher term. Message: Invalid argument: T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 for earlier term 2. Current term is 3. I1010 20:59:04.315052 3335 raft_consensus.cc:1654] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 for term 2 because replica is already servicing an update from a current leader or another vote. W1010 20:59:04.315052 3331 consensus_meta.cc:188] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 1.750s user 0.000s sys 0.000s I1010 20:59:04.315069 3331 raft_consensus.cc:1981] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.977s W1010 20:59:04.315094 3335 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.3:47101 (request call id 2) took 4701ms (client timeout 1672). W1010 20:59:04.315116 3335 rpcz_store.cc:238] Trace: 1010 20:58:59.613822 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:58:59.613829 (+ 7us) service_pool.cc:202] Handling call 1010 20:59:04.315075 (+4701246us) spinlock_profiling.cc:255] Waited 4.7 s on lock 0x3e5b6d0. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f3fb2fbf181 00007f3fb1ea747c ffffffffffffffff 1010 20:59:04.315078 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":10812791168} I1010 20:59:04.315104 2984 raft_consensus.cc:1609] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 4: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:59:04.315131 3334 raft_consensus.cc:1981] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.504s I1010 20:59:04.315145 3333 raft_consensus.cc:1609] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 4: Already voted for candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in this term. I1010 20:59:04.315264 3273 leader_election.cc:361] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 4 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c. Message: Invalid argument: T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 4: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:59:04.315301 2921 leader_election.cc:361] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 4 election: Vote denied by peer 7f623b00f5ae4b9c8e8940692d5dfd44. Message: Invalid argument: T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 4: Already voted for candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in this term. W1010 20:59:04.323143 3763 consensus_meta.cc:188] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 1.759s user 0.000s sys 0.000s I1010 20:59:04.323182 3763 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.659s I1010 20:59:04.323243 3763 raft_consensus.cc:391] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 5 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } W1010 20:59:04.323274 3834 consensus_meta.cc:188] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 3.056s user 0.000s sys 0.000s I1010 20:59:04.323304 3834 raft_consensus.cc:1981] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.835s I1010 20:59:04.323329 3834 raft_consensus.cc:391] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } W1010 20:59:04.328014 3764 consensus_meta.cc:188] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.314s user 0.000s sys 0.000s I1010 20:59:04.328038 3764 raft_consensus.cc:1981] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.771s I1010 20:59:04.328059 3764 raft_consensus.cc:391] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } W1010 20:59:04.348587 3101 consensus_meta.cc:188] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 1.780s user 0.000s sys 0.000s W1010 20:59:04.348594 2628 consensus_meta.cc:188] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 1.782s user 0.000s sys 0.000s I1010 20:59:04.348605 3101 raft_consensus.cc:1677] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1380a27d6b824575a3a5e139bee139f4 in term 4. I1010 20:59:04.348608 2628 raft_consensus.cc:1677] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1380a27d6b824575a3a5e139bee139f4 in term 3. W1010 20:59:04.348618 2628 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.6:47001 (request call id 3) took 1782ms (client timeout 1819). W1010 20:59:04.348619 3101 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.6:59112 (request call id 4) took 3325ms (client timeout 2048). W1010 20:59:04.348629 2628 rpcz_store.cc:238] Trace: 1010 20:59:02.566134 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:59:02.566147 (+ 13us) service_pool.cc:202] Handling call 1010 20:59:04.348614 (+1782467us) inbound_call.cc:130] Queueing success response Metrics: {} W1010 20:59:04.348631 3101 rpcz_store.cc:238] Trace: 1010 20:59:01.023417 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:59:01.023430 (+ 13us) service_pool.cc:202] Handling call 1010 20:59:04.348614 (+3325184us) inbound_call.cc:130] Queueing success response Metrics: {} I1010 20:59:04.348687 4193 raft_consensus.cc:1981] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.128s I1010 20:59:04.348767 4193 raft_consensus.cc:1823] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 1380a27d6b824575a3a5e139bee139f4 with higher term. Message: Invalid argument: T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 for earlier term 3. Current term is 4. I1010 20:59:04.348803 3154 leader_election.cc:347] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 3 election: Vote granted by peer 1031512f5120422e9e22a0afabd78f61 I1010 20:59:04.348814 3678 raft_consensus.cc:375] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:04.348824 3678 raft_consensus.cc:2031] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Advancing to term 5 W1010 20:59:04.348949 3842 consensus_meta.cc:188] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 2.544s user 0.000s sys 0.000s I1010 20:59:04.348969 3842 raft_consensus.cc:1981] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.857s I1010 20:59:04.348973 2632 raft_consensus.cc:1981] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.825s I1010 20:59:04.349315 3678 raft_consensus.cc:1981] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.818s I1010 20:59:04.349339 3678 raft_consensus.cc:391] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 5 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } I1010 20:59:04.349822 3842 raft_consensus.cc:391] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:59:04.520998 3894 raft_consensus.cc:375] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:04.521013 3894 raft_consensus.cc:2031] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Advancing to term 4 I1010 20:59:04.576264 3698 raft_consensus.cc:375] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:04.576278 3698 raft_consensus.cc:2031] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Advancing to term 4 I1010 20:59:04.651278 3889 raft_consensus.cc:375] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:04.651309 3889 raft_consensus.cc:2031] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 2 FOLLOWER]: Advancing to term 3 I1010 20:59:04.666074 3781 raft_consensus.cc:375] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:04.666088 3781 raft_consensus.cc:2031] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Advancing to term 4 I1010 20:59:04.687403 3882 raft_consensus.cc:375] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:04.687419 3882 raft_consensus.cc:2031] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Advancing to term 4 I1010 20:59:04.811518 2861 raft_consensus.cc:857] T 981834f5e88f4585b6420fca7ddcb289 P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Refusing update from remote peer 1380a27d6b824575a3a5e139bee139f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 1. (index mismatch) I1010 20:59:04.811630 4228 consensus_queue.cc:652] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [LEADER]: Connected to new peer: Peer: 8de6cfa864d64c58bf4eafb689517c93, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:59:04.811841 2628 raft_consensus.cc:857] T 981834f5e88f4585b6420fca7ddcb289 P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Refusing update from remote peer 1380a27d6b824575a3a5e139bee139f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 1. (index mismatch) I1010 20:59:04.811919 4228 consensus_queue.cc:652] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 [LEADER]: Connected to new peer: Peer: 1031512f5120422e9e22a0afabd78f61, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:59:04.813426 3789 raft_consensus.cc:375] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:04.813454 3789 raft_consensus.cc:2031] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 3 FOLLOWER]: Advancing to term 4 I1010 20:59:04.813905 3789 raft_consensus.cc:1981] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.059s I1010 20:59:04.813937 3789 raft_consensus.cc:391] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 4 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } W1010 20:59:04.940773 2514 consensus_meta.cc:188] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 0.632s user 0.000s sys 0.000s I1010 20:59:04.940807 2514 raft_consensus.cc:1677] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1380a27d6b824575a3a5e139bee139f4 in term 4. W1010 20:59:04.940804 3452 consensus_meta.cc:188] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 0.632s user 0.000s sys 0.000s W1010 20:59:04.940835 2514 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.6:59570 (request call id 4) took 3919ms (client timeout 2048). I1010 20:59:04.940839 3452 raft_consensus.cc:1677] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1380a27d6b824575a3a5e139bee139f4 in term 4. W1010 20:59:04.940852 2514 rpcz_store.cc:238] Trace: 1010 20:59:01.021483 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:59:01.021501 (+ 18us) service_pool.cc:202] Handling call 1010 20:59:04.940830 (+3919329us) inbound_call.cc:130] Queueing success response Metrics: {} W1010 20:59:04.940861 3452 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.6:42979 (request call id 6) took 3917ms (client timeout 1612). W1010 20:59:04.940879 3452 rpcz_store.cc:238] Trace: 1010 20:59:01.023113 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:59:01.023238 (+ 125us) service_pool.cc:202] Handling call 1010 20:59:04.940856 (+3917618us) inbound_call.cc:130] Queueing success response Metrics: {} I1010 20:59:04.940882 4194 raft_consensus.cc:1981] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.675s I1010 20:59:04.940932 4194 raft_consensus.cc:1823] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 1380a27d6b824575a3a5e139bee139f4 with higher term. Message: Service unavailable: T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 for term 3 because replica is already servicing an update from a current leader or another vote. I1010 20:59:04.940943 3754 raft_consensus.cc:375] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 4 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:04.940964 3754 raft_consensus.cc:2031] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 4 FOLLOWER]: Advancing to term 5 W1010 20:59:04.940973 3816 consensus_meta.cc:188] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 1.942s user 0.000s sys 0.000s I1010 20:59:04.940984 3891 raft_consensus.cc:375] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 4 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:04.940989 3891 raft_consensus.cc:2031] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 4 FOLLOWER]: Advancing to term 5 I1010 20:59:04.940991 3816 raft_consensus.cc:1981] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.512s I1010 20:59:04.941014 3816 raft_consensus.cc:391] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } I1010 20:59:05.049612 3848 raft_consensus.cc:375] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 4 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:05.049631 3848 raft_consensus.cc:2031] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 4 FOLLOWER]: Advancing to term 5 I1010 20:59:05.247915 3867 raft_consensus.cc:375] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:05.247934 3867 raft_consensus.cc:2031] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Advancing to term 4 W1010 20:59:05.740854 3331 consensus_meta.cc:188] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 1.426s user 0.000s sys 0.000s W1010 20:59:05.740874 3764 consensus_meta.cc:188] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.413s user 0.000s sys 0.000s I1010 20:59:05.740881 3331 raft_consensus.cc:1677] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8062d66fd284035ae3bc2567bc11e7c in term 4. W1010 20:59:05.740898 3331 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.4:46735 (request call id 6) took 3175ms (client timeout 1503). W1010 20:59:05.740914 3331 rpcz_store.cc:238] Trace: 1010 20:59:02.565169 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:59:02.565177 (+ 8us) service_pool.cc:202] Handling call 1010 20:59:05.740892 (+3175715us) inbound_call.cc:130] Queueing success response Metrics: {"tcmalloc_contention_cycles":4352} I1010 20:59:05.740947 3337 raft_consensus.cc:1654] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 for term 4 because replica is already servicing an update from a current leader or another vote. I1010 20:59:05.741004 3337 rpcz_store.cc:248] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.8:39867 (request call id 4) took 1430ms. Request Metrics: {"spinlock_wait_cycles":3289008512} I1010 20:59:05.741015 3764 leader_election.cc:223] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 4 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 I1010 20:59:05.741056 3764 leader_election.cc:223] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 4 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 W1010 20:59:05.741255 3334 consensus_meta.cc:188] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 1.426s user 0.000s sys 0.000s I1010 20:59:05.741266 3334 raft_consensus.cc:1677] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 580cbaf7092444dfbc6fb344da88d7b6 in term 4. W1010 20:59:05.741273 3334 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.9:42466 (request call id 24) took 1429ms (client timeout 1634). W1010 20:59:05.741282 3334 rpcz_store.cc:238] Trace: 1010 20:59:04.311692 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:59:04.311706 (+ 14us) service_pool.cc:202] Handling call 1010 20:59:05.741269 (+1429563us) inbound_call.cc:130] Queueing success response Metrics: {"tcmalloc_contention_cycles":768} I1010 20:59:05.741297 3870 raft_consensus.cc:375] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 4 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:05.741302 3870 raft_consensus.cc:2031] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 4 FOLLOWER]: Advancing to term 5 W1010 20:59:05.741487 3834 consensus_meta.cc:188] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 1.418s user 0.000s sys 0.000s I1010 20:59:05.741505 3390 leader_election.cc:361] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 4 election: Vote denied by peer 7f623b00f5ae4b9c8e8940692d5dfd44. Message: Service unavailable: T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 for term 4 because replica is already servicing an update from a current leader or another vote. I1010 20:59:05.741518 3390 leader_election.cc:248] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. I1010 20:59:05.741566 3834 leader_election.cc:223] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 3 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:59:05.741580 3834 leader_election.cc:223] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 3 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 W1010 20:59:05.741595 3889 consensus_meta.cc:188] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 1.090s user 0.000s sys 0.000s I1010 20:59:05.741621 3889 raft_consensus.cc:1981] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.910s I1010 20:59:05.741646 3889 raft_consensus.cc:391] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:59:05.741693 3507 leader_election.cc:347] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 4 election: Vote granted by peer 7f623b00f5ae4b9c8e8940692d5dfd44 I1010 20:59:05.741704 3507 leader_election.cc:248] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. W1010 20:59:05.741729 3754 consensus_meta.cc:188] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 0.801s user 0.000s sys 0.004s I1010 20:59:05.741742 3870 raft_consensus.cc:1981] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.946s I1010 20:59:05.741753 3754 raft_consensus.cc:1981] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.077s I1010 20:59:05.741765 3870 raft_consensus.cc:391] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 5 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:59:05.741778 3754 raft_consensus.cc:391] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 5 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } I1010 20:59:05.741971 4240 raft_consensus.cc:1981] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.924s W1010 20:59:05.741993 3789 consensus_meta.cc:188] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 0.928s user 0.000s sys 0.000s I1010 20:59:05.742002 4240 raft_consensus.cc:1823] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: None given I1010 20:59:05.742008 4241 raft_consensus.cc:1981] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.583s I1010 20:59:05.742019 4241 raft_consensus.cc:1861] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 4 FOLLOWER]: Leader election won for term 4 I1010 20:59:05.742090 3789 leader_election.cc:223] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 4 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:59:05.742106 3789 leader_election.cc:223] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 4 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:59:05.742136 4241 raft_consensus.cc:467] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [term 4 LEADER]: Becoming Leader. State: Replica: 580cbaf7092444dfbc6fb344da88d7b6, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:59:05.742167 4241 consensus_queue.cc:151] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 4, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } W1010 20:59:05.742169 2632 consensus_meta.cc:188] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 1.393s user 0.000s sys 0.004s I1010 20:59:05.742185 2632 raft_consensus.cc:1677] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 580cbaf7092444dfbc6fb344da88d7b6 in term 4. I1010 20:59:05.742183 2984 raft_consensus.cc:2031] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 3 FOLLOWER]: Advancing to term 4 W1010 20:59:05.742208 2632 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.9:55298 (request call id 4) took 1431ms (client timeout 1634). W1010 20:59:05.742238 2632 rpcz_store.cc:238] Trace: 1010 20:59:04.310730 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:59:04.310764 (+ 34us) service_pool.cc:202] Handling call 1010 20:59:05.742191 (+1431427us) inbound_call.cc:130] Queueing success response Metrics: {"tcmalloc_contention_cycles":768} I1010 20:59:05.742259 2861 raft_consensus.cc:2031] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Advancing to term 4 I1010 20:59:05.742305 3505 leader_election.cc:347] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 4 election: Vote granted by peer 1031512f5120422e9e22a0afabd78f61 W1010 20:59:05.742313 3678 consensus_meta.cc:188] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 1.393s user 0.004s sys 0.000s I1010 20:59:05.742410 3678 leader_election.cc:223] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 5 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 I1010 20:59:05.742429 3678 leader_election.cc:223] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 5 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 I1010 20:59:05.742516 3217 raft_consensus.cc:2031] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 4 FOLLOWER]: Advancing to term 5 I1010 20:59:05.742797 2403 catalog_manager.cc:2018] T 17a4ac4e4e164cebb4d88f59511cf926 reported consensus state change: term changed from 0 to 4, leader changed from to 580cbaf7092444dfbc6fb344da88d7b6 (127.6.150.9). New consensus state: current_term: 4 leader_uuid: "580cbaf7092444dfbc6fb344da88d7b6" config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } } W1010 20:59:05.746579 3722 consensus_meta.cc:188] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6: Time spent flushing consensus metadata: real 1.437s user 0.000s sys 0.000s W1010 20:59:05.746626 3891 consensus_meta.cc:188] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 0.806s user 0.004s sys 0.000s I1010 20:59:05.746644 3891 raft_consensus.cc:1981] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.566s I1010 20:59:05.746692 3722 leader_election.cc:223] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 4 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:59:05.746706 3722 leader_election.cc:223] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 4 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 W1010 20:59:05.746866 3817 consensus_meta.cc:188] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 1.436s user 0.000s sys 0.000s W1010 20:59:05.746911 3848 consensus_meta.cc:188] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 0.697s user 0.000s sys 0.000s I1010 20:59:05.746934 3848 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.301s I1010 20:59:05.746954 3817 leader_election.cc:223] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 3 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 I1010 20:59:05.746970 3817 leader_election.cc:223] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 3 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:59:05.746973 3848 raft_consensus.cc:391] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 5 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } I1010 20:59:05.747145 3867 raft_consensus.cc:1981] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.881s I1010 20:59:05.747179 3867 raft_consensus.cc:391] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 4 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } W1010 20:59:05.747262 3781 consensus_meta.cc:188] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 1.081s user 0.000s sys 0.000s I1010 20:59:05.747277 3781 raft_consensus.cc:1981] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.132s I1010 20:59:05.747301 3781 raft_consensus.cc:391] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 4 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:59:05.747321 2861 raft_consensus.cc:1981] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.033s W1010 20:59:05.747480 3763 consensus_meta.cc:188] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 1.424s user 0.000s sys 0.000s I1010 20:59:05.747550 2865 raft_consensus.cc:2031] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Advancing to term 4 I1010 20:59:05.747580 2985 raft_consensus.cc:1584] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 for earlier term 3. Current term is 4. I1010 20:59:05.747594 3891 raft_consensus.cc:391] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 5 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } I1010 20:59:05.747603 3763 leader_election.cc:223] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 5 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 I1010 20:59:05.747627 3763 leader_election.cc:223] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 5 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:59:05.747639 3337 raft_consensus.cc:1584] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 for earlier term 3. Current term is 4. W1010 20:59:05.747709 2802 leader_election.cc:333] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 3 election: Vote denied by peer 7f623b00f5ae4b9c8e8940692d5dfd44 with higher term. Message: Invalid argument: T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 for earlier term 3. Current term is 4. W1010 20:59:05.747714 2749 consensus_meta.cc:188] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 1.438s user 0.000s sys 0.004s I1010 20:59:05.747717 2802 leader_election.cc:336] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term I1010 20:59:05.747725 2749 raft_consensus.cc:1677] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1380a27d6b824575a3a5e139bee139f4 in term 4. W1010 20:59:05.747735 2749 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.6:49570 (request call id 6) took 4724ms (client timeout 1612). W1010 20:59:05.747747 2749 rpcz_store.cc:238] Trace: 1010 20:59:01.023144 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:59:01.023164 (+ 20us) service_pool.cc:202] Handling call 1010 20:59:05.747732 (+4724568us) inbound_call.cc:130] Queueing success response Metrics: {"tcmalloc_contention_cycles":4864} W1010 20:59:05.747936 3698 consensus_meta.cc:188] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 1.172s user 0.000s sys 0.000s I1010 20:59:05.747956 3698 raft_consensus.cc:1981] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.921s I1010 20:59:05.747975 3698 raft_consensus.cc:391] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } I1010 20:59:05.748109 3738 raft_consensus.cc:375] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:05.748116 3738 raft_consensus.cc:2031] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Advancing to term 5 I1010 20:59:05.748232 3453 raft_consensus.cc:2031] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 4 FOLLOWER]: Advancing to term 5 I1010 20:59:05.748280 3217 raft_consensus.cc:1981] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.553s W1010 20:59:05.748309 3894 consensus_meta.cc:188] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.227s user 0.000s sys 0.000s I1010 20:59:05.748325 3894 raft_consensus.cc:1981] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.649s I1010 20:59:05.748349 3894 raft_consensus.cc:391] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } W1010 20:59:05.748448 3842 consensus_meta.cc:188] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 1.399s user 0.000s sys 0.000s I1010 20:59:05.748533 2984 raft_consensus.cc:1981] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.154s I1010 20:59:05.748559 3842 leader_election.cc:223] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 3 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:59:05.748574 3842 leader_election.cc:223] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 3 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 I1010 20:59:05.748574 3101 raft_consensus.cc:1609] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 3: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. I1010 20:59:05.748718 2682 leader_election.cc:361] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 3 election: Vote denied by peer 33ef4317e50042ba95e517992caa6531. Message: Invalid argument: T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 3: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. I1010 20:59:05.748730 2749 raft_consensus.cc:1609] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 3: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:59:05.748802 3038 leader_election.cc:361] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 3 election: Vote denied by peer f6d15ccd64ec49be8396ed78e2a88377. Message: Invalid argument: T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 3: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. W1010 20:59:05.748867 2802 leader_election.cc:333] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 3 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c with higher term. Message: Invalid argument: T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 for earlier term 3. Current term is 4. I1010 20:59:05.905251 3808 raft_consensus.cc:375] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:05.905305 3808 raft_consensus.cc:2031] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Advancing to term 4 I1010 20:59:05.905541 3808 raft_consensus.cc:1981] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.697s I1010 20:59:05.905596 3808 raft_consensus.cc:391] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } W1010 20:59:06.091823 2921 leader_election.cc:275] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 8de6cfa864d64c58bf4eafb689517c93: Timed out: RequestConsensusVote RPC to 127.6.150.3:40117 timed out after 1.780s I1010 20:59:06.091855 2921 leader_election.cc:248] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. I1010 20:59:06.091918 3869 raft_consensus.cc:375] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:06.091953 3869 raft_consensus.cc:2031] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Advancing to term 5 I1010 20:59:06.147127 3815 raft_consensus.cc:375] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 3 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:06.147171 3815 raft_consensus.cc:2031] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 3 FOLLOWER]: Advancing to term 4 I1010 20:59:06.192283 3819 raft_consensus.cc:375] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:06.192299 3819 raft_consensus.cc:2031] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Advancing to term 4 I1010 20:59:06.242754 3331 raft_consensus.cc:857] T 17a4ac4e4e164cebb4d88f59511cf926 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 4 FOLLOWER]: Refusing update from remote peer 580cbaf7092444dfbc6fb344da88d7b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 4 index: 1. (index mismatch) I1010 20:59:06.242810 2632 raft_consensus.cc:857] T 17a4ac4e4e164cebb4d88f59511cf926 P 1031512f5120422e9e22a0afabd78f61 [term 4 FOLLOWER]: Refusing update from remote peer 580cbaf7092444dfbc6fb344da88d7b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 4 index: 1. (index mismatch) I1010 20:59:06.243122 4241 consensus_queue.cc:652] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [LEADER]: Connected to new peer: Peer: 7f623b00f5ae4b9c8e8940692d5dfd44, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:59:06.243214 4250 consensus_queue.cc:652] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 [LEADER]: Connected to new peer: Peer: 1031512f5120422e9e22a0afabd78f61, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:59:06.251988 3780 raft_consensus.cc:375] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:06.252005 3780 raft_consensus.cc:2031] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Advancing to term 4 W1010 20:59:06.284529 3816 consensus_meta.cc:188] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 1.344s user 0.000s sys 0.000s W1010 20:59:06.284541 3867 consensus_meta.cc:188] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 0.537s user 0.000s sys 0.000s I1010 20:59:06.284806 3816 leader_election.cc:223] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 3 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:59:06.284831 3816 leader_election.cc:223] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 3 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 I1010 20:59:06.284917 2985 raft_consensus.cc:1584] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 for earlier term 3. Current term is 4. W1010 20:59:06.285019 3390 leader_election.cc:333] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 3 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c with higher term. Message: Invalid argument: T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 for earlier term 3. Current term is 4. I1010 20:59:06.285027 3390 leader_election.cc:336] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term I1010 20:59:06.285040 3867 leader_election.cc:223] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 4 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:59:06.285068 3867 leader_election.cc:223] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 4 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:59:06.285073 2628 raft_consensus.cc:1609] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 4: Already voted for candidate 1031512f5120422e9e22a0afabd78f61 in this term. I1010 20:59:06.285145 2985 raft_consensus.cc:1609] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 4: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:59:06.285187 3451 raft_consensus.cc:1654] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 for term 4 because replica is already servicing an update from a current leader or another vote. I1010 20:59:06.285261 2568 leader_election.cc:361] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 4 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c. Message: Invalid argument: T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 4: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:59:06.285279 2567 leader_election.cc:361] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 4 election: Vote denied by peer 42e94fc3adbb419591836d222d946978. Message: Service unavailable: T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 for term 4 because replica is already servicing an update from a current leader or another vote. I1010 20:59:06.285300 2567 leader_election.cc:248] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. I1010 20:59:06.285301 2919 leader_election.cc:361] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 4 election: Vote denied by peer 1031512f5120422e9e22a0afabd78f61. Message: Invalid argument: T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 4: Already voted for candidate 1031512f5120422e9e22a0afabd78f61 in this term. I1010 20:59:06.285356 2628 raft_consensus.cc:1584] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 for earlier term 3. Current term is 4. I1010 20:59:06.285410 4258 raft_consensus.cc:1981] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.567s I1010 20:59:06.285419 4258 raft_consensus.cc:1823] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c with higher term. Message: Invalid argument: T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 for earlier term 3. Current term is 4. W1010 20:59:06.285424 3388 leader_election.cc:333] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 3 election: Vote denied by peer 1031512f5120422e9e22a0afabd78f61 with higher term. Message: Invalid argument: T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 for earlier term 3. Current term is 4. I1010 20:59:06.285451 3452 raft_consensus.cc:2031] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Advancing to term 4 W1010 20:59:06.285555 3894 consensus_meta.cc:188] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 0.537s user 0.004s sys 0.000s I1010 20:59:06.285678 3894 leader_election.cc:223] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 4 election: Requesting vote from peer 580cbaf7092444dfbc6fb344da88d7b6 I1010 20:59:06.285698 2986 raft_consensus.cc:1609] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 in current term 4: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:59:06.285728 3894 leader_election.cc:223] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 4 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:59:06.285895 3507 leader_election.cc:361] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 4 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c. Message: Invalid argument: T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 in current term 4: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. W1010 20:59:06.286022 3889 consensus_meta.cc:188] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 0.544s user 0.000s sys 0.000s W1010 20:59:06.286036 3882 consensus_meta.cc:188] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.599s user 0.000s sys 0.000s I1010 20:59:06.286059 3882 raft_consensus.cc:1981] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.819s I1010 20:59:06.286108 3882 raft_consensus.cc:391] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } I1010 20:59:06.286157 3889 leader_election.cc:223] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 3 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:59:06.286183 3334 raft_consensus.cc:1609] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 3: Already voted for candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in this term. I1010 20:59:06.286206 3889 leader_election.cc:223] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 3 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:59:06.286257 3337 raft_consensus.cc:1654] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 for term 3 because replica is already servicing an update from a current leader or another vote. I1010 20:59:06.286299 3101 raft_consensus.cc:1609] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 3: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. I1010 20:59:06.286344 3039 leader_election.cc:361] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 3 election: Vote denied by peer 7f623b00f5ae4b9c8e8940692d5dfd44. Message: Service unavailable: T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 for term 3 because replica is already servicing an update from a current leader or another vote. I1010 20:59:06.286348 2685 leader_election.cc:361] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 3 election: Vote denied by peer 7f623b00f5ae4b9c8e8940692d5dfd44. Message: Invalid argument: T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 3: Already voted for candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in this term. I1010 20:59:06.286362 3039 leader_election.cc:248] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I1010 20:59:06.286365 2685 leader_election.cc:248] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I1010 20:59:06.286391 3270 leader_election.cc:361] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 3 election: Vote denied by peer 33ef4317e50042ba95e517992caa6531. Message: Invalid argument: T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 3: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. I1010 20:59:06.286408 3569 raft_consensus.cc:1609] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 4: Already voted for candidate 580cbaf7092444dfbc6fb344da88d7b6 in this term. I1010 20:59:06.286489 2749 raft_consensus.cc:1609] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 3: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:59:06.286545 2921 leader_election.cc:361] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 4 election: Vote denied by peer 580cbaf7092444dfbc6fb344da88d7b6. Message: Invalid argument: T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 4: Already voted for candidate 580cbaf7092444dfbc6fb344da88d7b6 in this term. I1010 20:59:06.286592 3272 leader_election.cc:361] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 3 election: Vote denied by peer f6d15ccd64ec49be8396ed78e2a88377. Message: Invalid argument: T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in current term 3: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:59:06.286593 4259 raft_consensus.cc:1981] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.586s I1010 20:59:06.286602 3272 leader_election.cc:248] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I1010 20:59:06.286604 4259 raft_consensus.cc:1823] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: None given I1010 20:59:06.286715 4262 raft_consensus.cc:1981] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.929s W1010 20:59:06.286721 3870 consensus_meta.cc:188] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent flushing consensus metadata: real 0.545s user 0.000s sys 0.000s I1010 20:59:06.286728 4262 raft_consensus.cc:1823] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given I1010 20:59:06.286819 4260 raft_consensus.cc:1981] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.593s I1010 20:59:06.286845 4260 raft_consensus.cc:1823] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given I1010 20:59:06.286857 3870 leader_election.cc:223] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 5 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:59:06.286875 3870 leader_election.cc:223] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 5 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:59:06.286952 4261 raft_consensus.cc:1981] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.535s I1010 20:59:06.286989 3451 raft_consensus.cc:2031] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 4 FOLLOWER]: Advancing to term 5 I1010 20:59:06.286999 2986 raft_consensus.cc:2031] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Advancing to term 5 I1010 20:59:06.287009 4261 raft_consensus.cc:1823] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given W1010 20:59:06.288475 3891 consensus_meta.cc:188] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 0.541s user 0.000s sys 0.000s W1010 20:59:06.288497 3754 consensus_meta.cc:188] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 0.547s user 0.000s sys 0.000s W1010 20:59:06.288635 3453 consensus_meta.cc:188] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 0.540s user 0.004s sys 0.000s I1010 20:59:06.288653 3453 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.164s I1010 20:59:06.288810 3754 leader_election.cc:223] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 5 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:59:06.288847 3754 leader_election.cc:223] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 5 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 I1010 20:59:06.288961 3815 raft_consensus.cc:1981] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.975s I1010 20:59:06.289000 3815 raft_consensus.cc:391] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 4 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:59:06.289024 3101 raft_consensus.cc:1609] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 in current term 5: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. W1010 20:59:06.289156 3217 consensus_meta.cc:188] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 0.541s user 0.000s sys 0.000s I1010 20:59:06.289172 3217 raft_consensus.cc:1677] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33ef4317e50042ba95e517992caa6531 in term 5. I1010 20:59:06.289217 2447 leader_election.cc:361] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 5 election: Vote denied by peer 33ef4317e50042ba95e517992caa6531. Message: Invalid argument: T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 in current term 5: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. I1010 20:59:06.289265 2514 raft_consensus.cc:1609] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 5: Already voted for candidate 84bde34331b94c4ea003e48f9317f792 in this term. I1010 20:59:06.289309 3036 leader_election.cc:347] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 5 election: Vote granted by peer 1380a27d6b824575a3a5e139bee139f4 I1010 20:59:06.289329 3036 leader_election.cc:248] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. I1010 20:59:06.289361 3220 raft_consensus.cc:1654] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 for term 5 because replica is already servicing an update from a current leader or another vote. I1010 20:59:06.289402 3891 leader_election.cc:223] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 5 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:59:06.289409 3039 leader_election.cc:361] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 5 election: Vote denied by peer 84bde34331b94c4ea003e48f9317f792. Message: Invalid argument: T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 5: Already voted for candidate 84bde34331b94c4ea003e48f9317f792 in this term. I1010 20:59:06.289422 3891 leader_election.cc:223] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 5 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 W1010 20:59:06.289419 3848 consensus_meta.cc:188] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 0.542s user 0.000s sys 0.000s I1010 20:59:06.289430 2447 leader_election.cc:361] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 5 election: Vote denied by peer 1380a27d6b824575a3a5e139bee139f4. Message: Service unavailable: T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 for term 5 because replica is already servicing an update from a current leader or another vote. I1010 20:59:06.289448 2447 leader_election.cc:248] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. I1010 20:59:06.289564 3848 leader_election.cc:223] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 5 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:59:06.289578 3848 leader_election.cc:223] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 5 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:59:06.289605 3218 raft_consensus.cc:1609] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 5: Already voted for candidate 1380a27d6b824575a3a5e139bee139f4 in this term. I1010 20:59:06.289703 3220 raft_consensus.cc:2031] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 4 FOLLOWER]: Advancing to term 5 I1010 20:59:06.289710 4263 raft_consensus.cc:1981] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.279s I1010 20:59:06.289721 4263 raft_consensus.cc:1861] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 5 FOLLOWER]: Leader election won for term 5 I1010 20:59:06.289760 4264 raft_consensus.cc:1981] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.854s I1010 20:59:06.289767 4264 raft_consensus.cc:1823] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: None given I1010 20:59:06.289784 2745 raft_consensus.cc:1609] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 in current term 5: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:59:06.289877 2682 leader_election.cc:361] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 5 election: Vote denied by peer 1380a27d6b824575a3a5e139bee139f4. Message: Invalid argument: T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 5: Already voted for candidate 1380a27d6b824575a3a5e139bee139f4 in this term. I1010 20:59:06.289880 3155 leader_election.cc:361] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 5 election: Vote denied by peer f6d15ccd64ec49be8396ed78e2a88377. Message: Invalid argument: T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 in current term 5: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. W1010 20:59:06.289891 3698 consensus_meta.cc:188] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 0.542s user 0.000s sys 0.000s I1010 20:59:06.290007 3220 raft_consensus.cc:1981] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.212s I1010 20:59:06.290012 3698 leader_election.cc:223] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 4 election: Requesting vote from peer 580cbaf7092444dfbc6fb344da88d7b6 I1010 20:59:06.290032 3698 leader_election.cc:223] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 4 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:59:06.290055 2744 raft_consensus.cc:1609] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 in current term 4: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:59:06.290159 2749 raft_consensus.cc:1654] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c for term 4 because replica is already servicing an update from a current leader or another vote. I1010 20:59:06.290267 3506 leader_election.cc:361] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 4 election: Vote denied by peer f6d15ccd64ec49be8396ed78e2a88377. Message: Invalid argument: T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 580cbaf7092444dfbc6fb344da88d7b6 in current term 4: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:59:06.290277 3506 leader_election.cc:248] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. I1010 20:59:06.290316 2920 leader_election.cc:361] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 4 election: Vote denied by peer f6d15ccd64ec49be8396ed78e2a88377. Message: Service unavailable: T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c for term 4 because replica is already servicing an update from a current leader or another vote. I1010 20:59:06.290339 2920 leader_election.cc:248] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. I1010 20:59:06.290452 4266 raft_consensus.cc:1981] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.970s I1010 20:59:06.290463 4267 raft_consensus.cc:1981] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.090s I1010 20:59:06.290482 4266 raft_consensus.cc:1823] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: None given W1010 20:59:06.290480 3738 consensus_meta.cc:188] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 0.542s user 0.000s sys 0.000s I1010 20:59:06.290482 4263 raft_consensus.cc:467] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [term 5 LEADER]: Becoming Leader. State: Replica: 33ef4317e50042ba95e517992caa6531, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:59:06.290500 3738 raft_consensus.cc:1981] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.812s I1010 20:59:06.290500 4267 raft_consensus.cc:1823] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: None given I1010 20:59:06.290523 4263 consensus_queue.cc:151] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 5, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } I1010 20:59:06.290532 3738 raft_consensus.cc:391] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 5 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } I1010 20:59:06.291033 3819 raft_consensus.cc:1981] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.651s I1010 20:59:06.291064 3819 raft_consensus.cc:391] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } I1010 20:59:06.291193 2409 catalog_manager.cc:2018] T 7d680e6840854bca9daef77d506408e9 reported consensus state change: term changed from 0 to 5, leader changed from to 33ef4317e50042ba95e517992caa6531 (127.6.150.5). New consensus state: current_term: 5 leader_uuid: "33ef4317e50042ba95e517992caa6531" config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } } I1010 20:59:06.291405 2985 raft_consensus.cc:1609] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 4: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:59:06.291429 3569 raft_consensus.cc:1609] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 4: Already voted for candidate 580cbaf7092444dfbc6fb344da88d7b6 in this term. W1010 20:59:06.291581 3781 consensus_meta.cc:188] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 0.544s user 0.000s sys 0.000s I1010 20:59:06.291609 3452 raft_consensus.cc:1981] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.750s W1010 20:59:06.291646 2861 consensus_meta.cc:188] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 0.544s user 0.000s sys 0.000s I1010 20:59:06.291664 2861 raft_consensus.cc:1677] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1380a27d6b824575a3a5e139bee139f4 in term 4. I1010 20:59:06.291702 3781 leader_election.cc:223] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 4 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:59:06.291736 3781 leader_election.cc:223] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 4 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 I1010 20:59:06.291841 2685 leader_election.cc:361] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 4 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c. Message: Invalid argument: T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 4: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:59:06.291872 2685 leader_election.cc:361] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 4 election: Vote denied by peer 580cbaf7092444dfbc6fb344da88d7b6. Message: Invalid argument: T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 4: Already voted for candidate 580cbaf7092444dfbc6fb344da88d7b6 in this term. I1010 20:59:06.291870 3156 leader_election.cc:347] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 4 election: Vote granted by peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:59:06.291878 3156 leader_election.cc:248] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. I1010 20:59:06.291875 2685 leader_election.cc:248] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. I1010 20:59:06.291893 3451 raft_consensus.cc:1981] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.889s W1010 20:59:06.292096 2865 consensus_meta.cc:188] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 0.545s user 0.000s sys 0.000s I1010 20:59:06.292112 2865 raft_consensus.cc:1981] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.136s I1010 20:59:06.292131 4271 raft_consensus.cc:1981] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.767s I1010 20:59:06.292142 4271 raft_consensus.cc:1823] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: None given I1010 20:59:06.292426 4270 raft_consensus.cc:1981] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.684s I1010 20:59:06.292441 4270 raft_consensus.cc:1861] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 4 FOLLOWER]: Leader election won for term 4 I1010 20:59:06.292615 4270 raft_consensus.cc:467] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [term 4 LEADER]: Becoming Leader. State: Replica: 1380a27d6b824575a3a5e139bee139f4, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:59:06.292654 3452 raft_consensus.cc:1677] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8062d66fd284035ae3bc2567bc11e7c in term 4. I1010 20:59:06.292696 4270 consensus_queue.cc:151] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 4, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:59:06.292757 2920 leader_election.cc:347] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 4 election: Vote granted by peer 42e94fc3adbb419591836d222d946978 I1010 20:59:06.292768 2920 leader_election.cc:248] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. I1010 20:59:06.292773 3453 raft_consensus.cc:1677] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d15ccd64ec49be8396ed78e2a88377 in term 5. I1010 20:59:06.292804 3779 raft_consensus.cc:375] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 5 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:06.292829 3779 raft_consensus.cc:2031] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 5 FOLLOWER]: Advancing to term 6 I1010 20:59:06.292925 2684 leader_election.cc:347] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 5 election: Vote granted by peer 42e94fc3adbb419591836d222d946978 I1010 20:59:06.292928 4273 raft_consensus.cc:1981] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.680s I1010 20:59:06.292933 2684 leader_election.cc:248] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. I1010 20:59:06.292938 4273 raft_consensus.cc:1861] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Leader election won for term 4 I1010 20:59:06.293212 4275 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.182s I1010 20:59:06.293227 4273 raft_consensus.cc:467] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [term 4 LEADER]: Becoming Leader. State: Replica: b8062d66fd284035ae3bc2567bc11e7c, State: 1, Role: LEADER Watermarks: {Received: term: 2 index: 1 Committed: term: 2 index: 1} I1010 20:59:06.293251 4275 raft_consensus.cc:1861] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 5 FOLLOWER]: Leader election won for term 5 I1010 20:59:06.293315 4273 consensus_queue.cc:151] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Current term: 4, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } I1010 20:59:06.293342 2409 catalog_manager.cc:2018] T 949b26dfeadc4de38868bf4ed994d184 reported consensus state change: term changed from 0 to 4, leader changed from to 1380a27d6b824575a3a5e139bee139f4 (127.6.150.6). New consensus state: current_term: 4 leader_uuid: "1380a27d6b824575a3a5e139bee139f4" config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } } I1010 20:59:06.293390 4275 raft_consensus.cc:467] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 5 LEADER]: Becoming Leader. State: Replica: f6d15ccd64ec49be8396ed78e2a88377, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:59:06.293453 4275 consensus_queue.cc:151] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 5, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } I1010 20:59:06.293494 3815 leader_election.cc:223] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 4 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:59:06.293503 2514 raft_consensus.cc:1609] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 4: Already voted for candidate 84bde34331b94c4ea003e48f9317f792 in this term. I1010 20:59:06.293514 3815 leader_election.cc:223] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 4 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:59:06.293613 2802 leader_election.cc:361] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 4 election: Vote denied by peer 84bde34331b94c4ea003e48f9317f792. Message: Invalid argument: T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 4: Already voted for candidate 84bde34331b94c4ea003e48f9317f792 in this term. I1010 20:59:06.293622 2861 raft_consensus.cc:1609] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 in current term 4: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. I1010 20:59:06.293690 2450 leader_election.cc:361] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 4 election: Vote denied by peer 8de6cfa864d64c58bf4eafb689517c93. Message: Invalid argument: T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 in current term 4: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. I1010 20:59:06.293926 2409 catalog_manager.cc:2018] T 84fcfff9fdef4acf8c2b47f36330567f reported consensus state change: term changed from 2 to 4. New consensus state: current_term: 4 leader_uuid: "b8062d66fd284035ae3bc2567bc11e7c" config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } } I1010 20:59:06.295061 2409 catalog_manager.cc:2018] T 90fa7ae3643e4575a6be1c4936b6cc11 reported consensus state change: term changed from 0 to 5, leader changed from to f6d15ccd64ec49be8396ed78e2a88377 (127.6.150.2). New consensus state: current_term: 5 leader_uuid: "f6d15ccd64ec49be8396ed78e2a88377" config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } } W1010 20:59:06.409598 3273 leader_election.cc:275] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 8de6cfa864d64c58bf4eafb689517c93: Timed out: RequestConsensusVote RPC to 127.6.150.3:40117 timed out after 2.094s I1010 20:59:06.409615 3273 leader_election.cc:248] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. I1010 20:59:06.409759 4283 raft_consensus.cc:1981] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.593s I1010 20:59:06.409771 4283 raft_consensus.cc:1823] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: None given I1010 20:59:06.680902 3748 raft_consensus.cc:375] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:06.680915 3748 raft_consensus.cc:2031] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 3 FOLLOWER]: Advancing to term 4 I1010 20:59:06.681183 3748 raft_consensus.cc:1981] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.126s I1010 20:59:06.681215 3748 raft_consensus.cc:391] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 4 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } I1010 20:59:06.779067 3713 raft_consensus.cc:375] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:06.779081 3713 raft_consensus.cc:2031] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Advancing to term 4 I1010 20:59:06.791023 2514 raft_consensus.cc:857] T 7d680e6840854bca9daef77d506408e9 P 84bde34331b94c4ea003e48f9317f792 [term 5 FOLLOWER]: Refusing update from remote peer 33ef4317e50042ba95e517992caa6531: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 5 index: 1. (index mismatch) I1010 20:59:06.791074 3218 raft_consensus.cc:857] T 7d680e6840854bca9daef77d506408e9 P 1380a27d6b824575a3a5e139bee139f4 [term 5 FOLLOWER]: Refusing update from remote peer 33ef4317e50042ba95e517992caa6531: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 5 index: 1. (index mismatch) I1010 20:59:06.791153 4263 consensus_queue.cc:652] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [LEADER]: Connected to new peer: Peer: 84bde34331b94c4ea003e48f9317f792, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:59:06.791313 4263 consensus_queue.cc:652] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 [LEADER]: Connected to new peer: Peer: 1380a27d6b824575a3a5e139bee139f4, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:59:06.793424 2861 raft_consensus.cc:857] T 949b26dfeadc4de38868bf4ed994d184 P 8de6cfa864d64c58bf4eafb689517c93 [term 4 FOLLOWER]: Refusing update from remote peer 1380a27d6b824575a3a5e139bee139f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 4 index: 1. (index mismatch) I1010 20:59:06.793548 4270 consensus_queue.cc:652] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [LEADER]: Connected to new peer: Peer: 8de6cfa864d64c58bf4eafb689517c93, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:59:06.793740 3218 raft_consensus.cc:857] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 5 FOLLOWER]: Refusing update from remote peer f6d15ccd64ec49be8396ed78e2a88377: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 5 index: 1. (index mismatch) I1010 20:59:06.793776 2628 raft_consensus.cc:857] T 84fcfff9fdef4acf8c2b47f36330567f P 1031512f5120422e9e22a0afabd78f61 [term 4 FOLLOWER]: Refusing update from remote peer b8062d66fd284035ae3bc2567bc11e7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 4 index: 2. (index mismatch) I1010 20:59:06.793877 4273 consensus_queue.cc:652] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [LEADER]: Connected to new peer: Peer: 1031512f5120422e9e22a0afabd78f61, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:59:06.793925 3452 raft_consensus.cc:857] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 4 FOLLOWER]: Refusing update from remote peer b8062d66fd284035ae3bc2567bc11e7c: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch) I1010 20:59:06.793949 4275 consensus_queue.cc:652] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [LEADER]: Connected to new peer: Peer: 1380a27d6b824575a3a5e139bee139f4, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:59:06.794023 4273 consensus_queue.cc:652] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c [LEADER]: Connected to new peer: Peer: 42e94fc3adbb419591836d222d946978, Is new: false, Last received: 2.1, Next index: 2, Last known committed idx: 1, Last exchange result: ERROR, Needs tablet copy: false W1010 20:59:07.456557 3808 consensus_meta.cc:188] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 1.551s user 0.000s sys 0.000s I1010 20:59:07.456810 3808 leader_election.cc:223] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 4 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 I1010 20:59:07.456825 3808 leader_election.cc:223] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 4 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:59:07.457056 3101 raft_consensus.cc:1609] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 4: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. W1010 20:59:07.457061 3780 consensus_meta.cc:188] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 1.205s user 0.000s sys 0.000s I1010 20:59:07.457062 2514 raft_consensus.cc:1609] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 4: Already voted for candidate 84bde34331b94c4ea003e48f9317f792 in this term. I1010 20:59:07.457082 3780 raft_consensus.cc:1981] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.859s I1010 20:59:07.457111 3780 raft_consensus.cc:391] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } I1010 20:59:07.457128 3101 rpcz_store.cc:248] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.3:53121 (request call id 4) took 1164ms. Request Metrics: {"spinlock_wait_cycles":2678609152} W1010 20:59:07.457137 3713 consensus_meta.cc:188] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 0.678s user 0.000s sys 0.000s I1010 20:59:07.457159 3039 leader_election.cc:361] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 4 election: Vote denied by peer 84bde34331b94c4ea003e48f9317f792. Message: Invalid argument: T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 4: Already voted for candidate 84bde34331b94c4ea003e48f9317f792 in this term. I1010 20:59:07.457165 3713 raft_consensus.cc:1981] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.740s I1010 20:59:07.457190 3713 raft_consensus.cc:391] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } W1010 20:59:07.457216 3882 consensus_meta.cc:188] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.171s user 0.000s sys 0.000s I1010 20:59:07.457403 3882 leader_election.cc:223] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 4 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:59:07.457422 3882 leader_election.cc:223] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 4 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 W1010 20:59:07.457571 2984 consensus_meta.cc:188] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.709s user 0.000s sys 0.000s I1010 20:59:07.457595 2984 raft_consensus.cc:1677] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1380a27d6b824575a3a5e139bee139f4 in term 4. W1010 20:59:07.457617 2984 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.6:46028 (request call id 4) took 1715ms (client timeout 2059). W1010 20:59:07.457638 2984 rpcz_store.cc:238] Trace: 1010 20:59:05.742164 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:59:05.742172 (+ 8us) service_pool.cc:202] Handling call 1010 20:59:07.457613 (+1715441us) inbound_call.cc:130] Queueing success response Metrics: {} I1010 20:59:07.457653 2985 raft_consensus.cc:857] T 949b26dfeadc4de38868bf4ed994d184 P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Refusing update from remote peer 1380a27d6b824575a3a5e139bee139f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 4 index: 1. (index mismatch) I1010 20:59:07.457923 3156 leader_election.cc:347] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 4 election: Vote granted by peer b8062d66fd284035ae3bc2567bc11e7c W1010 20:59:07.457937 3869 consensus_meta.cc:188] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.366s user 0.000s sys 0.000s I1010 20:59:07.457978 3869 raft_consensus.cc:1981] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.728s I1010 20:59:07.458008 3869 raft_consensus.cc:391] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 5 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:59:07.458050 4288 consensus_queue.cc:652] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 [LEADER]: Connected to new peer: Peer: b8062d66fd284035ae3bc2567bc11e7c, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false W1010 20:59:07.458081 2986 consensus_meta.cc:188] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.171s user 0.000s sys 0.000s I1010 20:59:07.458151 2986 raft_consensus.cc:1981] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.821s W1010 20:59:07.458232 3748 consensus_meta.cc:188] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61: Time spent flushing consensus metadata: real 0.777s user 0.000s sys 0.000s I1010 20:59:07.458341 2799 leader_election.cc:361] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 4 election: Vote denied by peer 33ef4317e50042ba95e517992caa6531. Message: Invalid argument: T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 4: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. I1010 20:59:07.458359 2799 leader_election.cc:248] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. I1010 20:59:07.458385 3748 leader_election.cc:223] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 4 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:59:07.458400 3748 leader_election.cc:223] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 4 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:59:07.458410 2861 raft_consensus.cc:1609] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 4: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. I1010 20:59:07.458487 3039 leader_election.cc:361] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 4 election: Vote denied by peer 8de6cfa864d64c58bf4eafb689517c93. Message: Invalid argument: T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 4: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. I1010 20:59:07.458494 3039 leader_election.cc:248] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. I1010 20:59:07.458611 4300 raft_consensus.cc:1981] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.932s I1010 20:59:07.458612 4299 raft_consensus.cc:1981] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.509s I1010 20:59:07.458627 4299 raft_consensus.cc:1823] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: None given I1010 20:59:07.458642 3099 raft_consensus.cc:1654] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 for term 4 because replica is already servicing an update from a current leader or another vote. I1010 20:59:07.458672 2514 raft_consensus.cc:2031] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 3 FOLLOWER]: Advancing to term 4 I1010 20:59:07.458724 3099 rpcz_store.cc:248] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.0:46745 (request call id 10) took 1165ms. Request Metrics: {"spinlock_wait_cycles":2679631616} I1010 20:59:07.458775 4300 raft_consensus.cc:1823] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: None given I1010 20:59:07.458783 2447 leader_election.cc:361] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 4 election: Vote denied by peer 33ef4317e50042ba95e517992caa6531. Message: Service unavailable: T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bde34331b94c4ea003e48f9317f792 for term 4 because replica is already servicing an update from a current leader or another vote. I1010 20:59:07.458791 2447 leader_election.cc:248] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. I1010 20:59:07.458932 4302 raft_consensus.cc:1981] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.142s I1010 20:59:07.458945 4302 raft_consensus.cc:1823] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: None given I1010 20:59:07.458950 2514 raft_consensus.cc:1981] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.076s I1010 20:59:07.468894 3881 raft_consensus.cc:375] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:07.468907 3881 raft_consensus.cc:2031] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 3 FOLLOWER]: Advancing to term 4 I1010 20:59:07.625851 3729 raft_consensus.cc:375] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:07.625865 3729 raft_consensus.cc:2031] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Advancing to term 4 W1010 20:59:07.748522 3220 consensus_meta.cc:188] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4: Time spent flushing consensus metadata: real 1.458s user 0.000s sys 0.000s I1010 20:59:07.748569 3220 raft_consensus.cc:1677] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42e94fc3adbb419591836d222d946978 in term 5. W1010 20:59:07.748582 3220 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.8:57074 (request call id 7) took 1458ms (client timeout 1566). W1010 20:59:07.748597 3220 rpcz_store.cc:238] Trace: 1010 20:59:06.289611 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:59:06.289688 (+ 77us) service_pool.cc:202] Handling call 1010 20:59:07.748577 (+1458889us) inbound_call.cc:130] Queueing success response Metrics: {} I1010 20:59:07.748775 3387 leader_election.cc:347] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 5 election: Vote granted by peer 1380a27d6b824575a3a5e139bee139f4 I1010 20:59:07.748788 3387 leader_election.cc:248] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. I1010 20:59:07.748910 4306 raft_consensus.cc:1981] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.817s I1010 20:59:07.748929 4306 raft_consensus.cc:1861] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 5 FOLLOWER]: Leader election won for term 5 I1010 20:59:07.749171 4306 raft_consensus.cc:467] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 5 LEADER]: Becoming Leader. State: Replica: 42e94fc3adbb419591836d222d946978, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:59:07.749236 4306 consensus_queue.cc:151] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 5, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } W1010 20:59:07.749433 3819 consensus_meta.cc:188] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 1.458s user 0.000s sys 0.000s W1010 20:59:07.749588 3738 consensus_meta.cc:188] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 1.459s user 0.000s sys 0.000s I1010 20:59:07.749691 3819 leader_election.cc:223] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 4 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:59:07.749701 2749 raft_consensus.cc:1609] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 4: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:59:07.749722 3819 leader_election.cc:223] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 4 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 I1010 20:59:07.749754 3738 leader_election.cc:223] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 5 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 I1010 20:59:07.749764 3738 leader_election.cc:223] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 5 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:59:07.749784 2748 raft_consensus.cc:1609] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 5: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. W1010 20:59:07.749814 2748 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.8:42100 (request call id 10) took 1460ms (client timeout 1566). I1010 20:59:07.749821 3220 raft_consensus.cc:1609] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 5: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. W1010 20:59:07.749828 2748 rpcz_store.cc:238] Trace: 1010 20:59:06.289691 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:59:06.289700 (+ 9us) service_pool.cc:202] Handling call 1010 20:59:07.749807 (+1460107us) spinlock_profiling.cc:255] Waited 1.46 s on lock 0x3419c10. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f7357684181 00007f735656c47c ffffffffffffffff 1010 20:59:07.749810 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3358171520} I1010 20:59:07.749864 3729 raft_consensus.cc:1981] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.839s I1010 20:59:07.749887 3729 raft_consensus.cc:391] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } I1010 20:59:07.749912 2567 leader_election.cc:361] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 4 election: Vote denied by peer f6d15ccd64ec49be8396ed78e2a88377. Message: Invalid argument: T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 4: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:59:07.749968 3389 leader_election.cc:361] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 5 election: Vote denied by peer f6d15ccd64ec49be8396ed78e2a88377. Message: Invalid argument: T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 5: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:59:07.750010 2682 leader_election.cc:361] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 5 election: Vote denied by peer 1380a27d6b824575a3a5e139bee139f4. Message: Invalid argument: T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 5: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:59:07.749996 2628 raft_consensus.cc:1609] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 4: Already voted for candidate 1031512f5120422e9e22a0afabd78f61 in this term. I1010 20:59:07.750121 2683 leader_election.cc:361] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 4 election: Vote denied by peer 1031512f5120422e9e22a0afabd78f61. Message: Invalid argument: T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 4: Already voted for candidate 1031512f5120422e9e22a0afabd78f61 in this term. I1010 20:59:07.750232 3452 raft_consensus.cc:1640] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 5 LEADER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 for term 5 because replica is either leader or believes a valid leader to be alive. I1010 20:59:07.750321 2684 leader_election.cc:361] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 5 election: Vote denied by peer 42e94fc3adbb419591836d222d946978. Message: Invalid argument: T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [term 5 LEADER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 for term 5 because replica is either leader or believes a valid leader to be alive. I1010 20:59:07.750334 2684 leader_election.cc:248] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. I1010 20:59:07.750427 2402 catalog_manager.cc:2018] T f689db1eddb54b1f85b7df38d14534b4 reported consensus state change: term changed from 0 to 5, leader changed from to 42e94fc3adbb419591836d222d946978 (127.6.150.8). New consensus state: current_term: 5 leader_uuid: "42e94fc3adbb419591836d222d946978" config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } } I1010 20:59:07.750514 4310 raft_consensus.cc:1981] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.263s I1010 20:59:07.750529 4310 raft_consensus.cc:1823] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: None given W1010 20:59:07.750777 3451 consensus_meta.cc:188] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 1.459s user 0.000s sys 0.000s I1010 20:59:07.750793 3451 raft_consensus.cc:1677] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in term 5. W1010 20:59:07.750803 3451 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.7:59528 (request call id 4) took 1463ms (client timeout 1946). W1010 20:59:07.750815 3451 rpcz_store.cc:238] Trace: 1010 20:59:06.286961 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:59:06.286972 (+ 11us) service_pool.cc:202] Handling call 1010 20:59:07.750799 (+1463827us) inbound_call.cc:130] Queueing success response Metrics: {"tcmalloc_contention_cycles":384} W1010 20:59:07.750856 3779 consensus_meta.cc:188] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 1.458s user 0.000s sys 0.000s I1010 20:59:07.750892 3779 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.784s I1010 20:59:07.750895 3272 leader_election.cc:347] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 5 election: Vote granted by peer 42e94fc3adbb419591836d222d946978 I1010 20:59:07.750905 3272 leader_election.cc:248] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. I1010 20:59:07.750926 3779 raft_consensus.cc:391] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 6 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } I1010 20:59:07.751107 4311 raft_consensus.cc:1981] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.972s I1010 20:59:07.751127 4311 raft_consensus.cc:1861] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 5 FOLLOWER]: Leader election won for term 5 I1010 20:59:07.751236 3881 raft_consensus.cc:1981] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.562s I1010 20:59:07.751286 3881 raft_consensus.cc:391] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 4 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } I1010 20:59:07.751284 4311 raft_consensus.cc:467] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 5 LEADER]: Becoming Leader. State: Replica: 7f623b00f5ae4b9c8e8940692d5dfd44, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:59:07.751340 4311 consensus_queue.cc:151] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 5, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } W1010 20:59:07.752084 2865 consensus_meta.cc:188] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 1.460s user 0.000s sys 0.000s I1010 20:59:07.752112 2402 catalog_manager.cc:2018] T f2289fb138ea4a5ab6c02803c4e9aa51 reported consensus state change: term changed from 0 to 5, leader changed from to 7f623b00f5ae4b9c8e8940692d5dfd44 (127.6.150.7). New consensus state: current_term: 5 leader_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } } I1010 20:59:07.752120 2865 raft_consensus.cc:1677] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8062d66fd284035ae3bc2567bc11e7c in term 4. I1010 20:59:07.752177 2862 raft_consensus.cc:1654] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f623b00f5ae4b9c8e8940692d5dfd44 for term 4 because replica is already servicing an update from a current leader or another vote. W1010 20:59:07.752182 2865 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.4:46161 (request call id 7) took 3441ms (client timeout 1780). W1010 20:59:07.752267 2862 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.7:50821 (request call id 3) took 3437ms (client timeout 2094). W1010 20:59:07.752270 2865 rpcz_store.cc:238] Trace: 1010 20:59:04.310823 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:59:04.310847 (+ 24us) service_pool.cc:202] Handling call 1010 20:59:07.752174 (+3441327us) spinlock_profiling.cc:255] Waited 1.44 s on lock 0x3f9e470. stack: 00000000019065e1 0000000000976813 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f9e2c5b9181 00007f9e2b4a147c ffffffffffffffff 1010 20:59:07.752178 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":3304392064} W1010 20:59:07.752281 2862 rpcz_store.cc:238] Trace: 1010 20:59:04.315182 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:59:04.315193 (+ 11us) service_pool.cc:202] Handling call 1010 20:59:07.752259 (+3437066us) spinlock_profiling.cc:255] Waited 3.44 s on lock 0x3f9e470. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007f9e2c5b9181 00007f9e2b4a147c ffffffffffffffff 1010 20:59:07.752263 (+ 4us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":7904997632} I1010 20:59:07.752370 4248 raft_consensus.cc:1981] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.968s I1010 20:59:07.752396 4248 raft_consensus.cc:1823] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 7f623b00f5ae4b9c8e8940692d5dfd44 with higher term. Message: Invalid argument: T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 for earlier term 3. Current term is 4. I1010 20:59:07.752774 2986 raft_consensus.cc:1677] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in term 5. W1010 20:59:07.752804 2986 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.7:55183 (request call id 7) took 1465ms (client timeout 1946). W1010 20:59:07.752825 2986 rpcz_store.cc:238] Trace: 1010 20:59:06.286973 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:59:06.286992 (+ 19us) service_pool.cc:202] Handling call 1010 20:59:07.752799 (+1465807us) inbound_call.cc:130] Queueing success response Metrics: {} I1010 20:59:07.752915 3273 leader_election.cc:347] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 5 election: Vote granted by peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:59:07.769038 3671 raft_consensus.cc:375] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:07.769052 3671 raft_consensus.cc:2031] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 3 FOLLOWER]: Advancing to term 4 I1010 20:59:07.769243 3671 raft_consensus.cc:1981] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.963s I1010 20:59:07.769273 3671 raft_consensus.cc:391] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 4 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } W1010 20:59:07.794904 2684 consensus_peers.cc:332] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 -> Peer 42e94fc3adbb419591836d222d946978 (127.6.150.8:35299): Couldn't send request to peer 42e94fc3adbb419591836d222d946978 for tablet 90fa7ae3643e4575a6be1c4936b6cc11. Status: Timed out: UpdateConsensus RPC to 127.6.150.8:35299 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 1 times. I1010 20:59:08.249727 2748 raft_consensus.cc:857] T f689db1eddb54b1f85b7df38d14534b4 P f6d15ccd64ec49be8396ed78e2a88377 [term 5 FOLLOWER]: Refusing update from remote peer 42e94fc3adbb419591836d222d946978: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 5 index: 1. (index mismatch) I1010 20:59:08.249845 4306 consensus_queue.cc:652] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [LEADER]: Connected to new peer: Peer: f6d15ccd64ec49be8396ed78e2a88377, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:59:08.250378 3220 raft_consensus.cc:857] T f689db1eddb54b1f85b7df38d14534b4 P 1380a27d6b824575a3a5e139bee139f4 [term 5 FOLLOWER]: Refusing update from remote peer 42e94fc3adbb419591836d222d946978: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 5 index: 1. (index mismatch) I1010 20:59:08.250744 4306 consensus_queue.cc:652] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 [LEADER]: Connected to new peer: Peer: 1380a27d6b824575a3a5e139bee139f4, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:59:08.251727 3451 raft_consensus.cc:857] T f2289fb138ea4a5ab6c02803c4e9aa51 P 42e94fc3adbb419591836d222d946978 [term 5 FOLLOWER]: Refusing update from remote peer 7f623b00f5ae4b9c8e8940692d5dfd44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 5 index: 1. (index mismatch) I1010 20:59:08.251798 2986 raft_consensus.cc:857] T f2289fb138ea4a5ab6c02803c4e9aa51 P b8062d66fd284035ae3bc2567bc11e7c [term 5 FOLLOWER]: Refusing update from remote peer 7f623b00f5ae4b9c8e8940692d5dfd44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 5 index: 1. (index mismatch) I1010 20:59:08.251866 4311 consensus_queue.cc:652] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [LEADER]: Connected to new peer: Peer: 42e94fc3adbb419591836d222d946978, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:59:08.251991 4311 consensus_queue.cc:652] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 [LEADER]: Connected to new peer: Peer: b8062d66fd284035ae3bc2567bc11e7c, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false W1010 20:59:08.559175 3780 consensus_meta.cc:188] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 1.102s user 0.000s sys 0.004s I1010 20:59:08.559486 3780 leader_election.cc:223] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 4 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c W1010 20:59:08.559499 3713 consensus_meta.cc:188] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531: Time spent flushing consensus metadata: real 1.102s user 0.000s sys 0.004s I1010 20:59:08.559535 3780 leader_election.cc:223] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 4 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 I1010 20:59:08.559536 3101 raft_consensus.cc:1609] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 4: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. I1010 20:59:08.559612 3101 rpcz_store.cc:248] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.4:56288 (request call id 4) took 1101ms. Request Metrics: {"spinlock_wait_cycles":2532127104} I1010 20:59:08.559702 3713 leader_election.cc:223] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 4 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:59:08.559722 3713 leader_election.cc:223] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 4 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 I1010 20:59:08.559725 2986 raft_consensus.cc:1609] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 4: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. W1010 20:59:08.559763 3869 consensus_meta.cc:188] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c: Time spent flushing consensus metadata: real 1.102s user 0.000s sys 0.004s I1010 20:59:08.559767 2918 leader_election.cc:361] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 4 election: Vote denied by peer 33ef4317e50042ba95e517992caa6531. Message: Invalid argument: T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b8062d66fd284035ae3bc2567bc11e7c in current term 4: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. I1010 20:59:08.559811 2748 raft_consensus.cc:1609] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 4: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:59:08.559837 2628 raft_consensus.cc:1609] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 4: Already voted for candidate 1031512f5120422e9e22a0afabd78f61 in this term. I1010 20:59:08.559854 3039 leader_election.cc:361] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 4 election: Vote denied by peer b8062d66fd284035ae3bc2567bc11e7c. Message: Invalid argument: T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 4: Already voted for candidate b8062d66fd284035ae3bc2567bc11e7c in this term. I1010 20:59:08.559872 3869 leader_election.cc:223] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 5 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 I1010 20:59:08.559880 3038 leader_election.cc:361] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 4 election: Vote denied by peer f6d15ccd64ec49be8396ed78e2a88377. Message: Invalid argument: T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 4: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:59:08.559895 3869 leader_election.cc:223] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 5 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:59:08.559901 3037 leader_election.cc:361] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 4 election: Vote denied by peer 1031512f5120422e9e22a0afabd78f61. Message: Invalid argument: T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 in current term 4: Already voted for candidate 1031512f5120422e9e22a0afabd78f61 in this term. I1010 20:59:08.559909 3037 leader_election.cc:248] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. I1010 20:59:08.559926 4249 raft_consensus.cc:1981] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.775s I1010 20:59:08.559965 4249 raft_consensus.cc:1823] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 5 FOLLOWER]: Leader election lost for term 4. Reason: None given I1010 20:59:08.560021 3337 raft_consensus.cc:2031] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 4 FOLLOWER]: Advancing to term 5 I1010 20:59:08.560061 3102 raft_consensus.cc:1609] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 4: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. I1010 20:59:08.560134 3099 raft_consensus.cc:1654] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 for term 4 because replica is already servicing an update from a current leader or another vote. I1010 20:59:08.560166 3102 rpcz_store.cc:248] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.1:45293 (request call id 4) took 1101ms. Request Metrics: {"spinlock_wait_cycles":2533609856} I1010 20:59:08.560300 2682 leader_election.cc:361] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 4 election: Vote denied by peer 33ef4317e50042ba95e517992caa6531. Message: Service unavailable: T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 for term 4 because replica is already servicing an update from a current leader or another vote. I1010 20:59:08.560303 2565 leader_election.cc:361] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 4 election: Vote denied by peer 33ef4317e50042ba95e517992caa6531. Message: Invalid argument: T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1031512f5120422e9e22a0afabd78f61 in current term 4: Already voted for candidate 33ef4317e50042ba95e517992caa6531 in this term. I1010 20:59:08.560323 2682 leader_election.cc:248] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. I1010 20:59:08.560324 2565 leader_election.cc:248] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. I1010 20:59:08.560396 4322 raft_consensus.cc:1981] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.761s I1010 20:59:08.560416 4322 raft_consensus.cc:1823] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: None given I1010 20:59:08.560473 4323 raft_consensus.cc:1981] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.029s I1010 20:59:08.560503 4323 raft_consensus.cc:1823] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: None given I1010 20:59:08.560506 4324 raft_consensus.cc:1981] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.144s I1010 20:59:08.560539 4324 raft_consensus.cc:1823] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: None given I1010 20:59:08.560556 3337 raft_consensus.cc:1981] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.374s W1010 20:59:08.560856 2514 consensus_meta.cc:188] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792: Time spent flushing consensus metadata: real 1.102s user 0.000s sys 0.000s I1010 20:59:08.560907 2514 raft_consensus.cc:1677] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8062d66fd284035ae3bc2567bc11e7c in term 4. I1010 20:59:08.560961 2514 rpcz_store.cc:248] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.4:40901 (request call id 30) took 1102ms. Request Metrics: {} I1010 20:59:08.560999 2508 raft_consensus.cc:1654] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 for term 4 because replica is already servicing an update from a current leader or another vote. I1010 20:59:08.561103 2921 leader_election.cc:347] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 4 election: Vote granted by peer 84bde34331b94c4ea003e48f9317f792 I1010 20:59:08.561136 2921 leader_election.cc:248] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. I1010 20:59:08.561240 3039 leader_election.cc:361] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 4 election: Vote denied by peer 84bde34331b94c4ea003e48f9317f792. Message: Service unavailable: T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ef4317e50042ba95e517992caa6531 for term 4 because replica is already servicing an update from a current leader or another vote. I1010 20:59:08.561261 3039 leader_election.cc:248] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. I1010 20:59:08.561261 2862 raft_consensus.cc:2031] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 4 FOLLOWER]: Advancing to term 5 I1010 20:59:08.561504 4325 raft_consensus.cc:1981] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.593s I1010 20:59:08.561509 4326 raft_consensus.cc:1981] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.705s I1010 20:59:08.561519 4326 raft_consensus.cc:1823] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: None given I1010 20:59:08.561522 4325 raft_consensus.cc:1861] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Leader election won for term 4 I1010 20:59:08.561699 4325 raft_consensus.cc:467] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [term 4 LEADER]: Becoming Leader. State: Replica: b8062d66fd284035ae3bc2567bc11e7c, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:59:08.561743 4325 consensus_queue.cc:151] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 4, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } I1010 20:59:08.562269 2402 catalog_manager.cc:2018] T 8da5904fd56d441ea29e61eb9d2b0b2f reported consensus state change: term changed from 0 to 4, leader changed from to b8062d66fd284035ae3bc2567bc11e7c (127.6.150.4). New consensus state: current_term: 4 leader_uuid: "b8062d66fd284035ae3bc2567bc11e7c" config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } } W1010 20:59:08.563077 3779 consensus_meta.cc:188] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 0.812s user 0.000s sys 0.000s W1010 20:59:08.563117 3729 consensus_meta.cc:188] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377: Time spent flushing consensus metadata: real 0.813s user 0.004s sys 0.000s I1010 20:59:08.563253 3779 leader_election.cc:223] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 6 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 I1010 20:59:08.563258 3453 raft_consensus.cc:825] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 6 FOLLOWER]: Rejecting Update request from peer f6d15ccd64ec49be8396ed78e2a88377 for earlier term 5. Current term is 6. Ops: [] I1010 20:59:08.563280 3779 leader_election.cc:223] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 6 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:59:08.563282 3729 leader_election.cc:223] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 4 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:59:08.563328 3729 leader_election.cc:223] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 4 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 W1010 20:59:08.563375 3453 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.6.150.2:40678 (request call id 9) took 1769ms (client timeout 1000). W1010 20:59:08.563395 3453 rpcz_store.cc:238] Trace: 1010 20:59:06.793860 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:59:06.793867 (+ 7us) service_pool.cc:202] Handling call 1010 20:59:06.793873 (+ 6us) raft_consensus.cc:1068] Updating replica for 0 ops 1010 20:59:08.563282 (+1769409us) raft_consensus.cc:1270] Filling consensus response to leader. 1010 20:59:08.563363 (+ 81us) spinlock_profiling.cc:255] Waited 1.77 s on lock 0x3e4f970. stack: 00000000019065e1 0000000000978c4a 000000000097a5b3 00000000007880ea 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007fd760afa181 00007fd75f9e247c ffffffffffffffff 1010 20:59:08.563371 (+ 8us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":4069540224} I1010 20:59:08.563423 3454 raft_consensus.cc:1654] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 for term 5 because replica is already servicing an update from a current leader or another vote. I1010 20:59:08.563436 3220 raft_consensus.cc:1640] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 for term 6 because replica is either leader or believes a valid leader to be alive. W1010 20:59:08.563465 3454 rpcz_store.cc:234] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.6.150.6:42979 (request call id 7) took 2273ms (client timeout 2301). W1010 20:59:08.563477 3454 rpcz_store.cc:238] Trace: 1010 20:59:06.289825 (+ 0us) service_pool.cc:143] Inserting onto call queue 1010 20:59:06.289833 (+ 8us) service_pool.cc:202] Handling call 1010 20:59:08.563460 (+2273627us) spinlock_profiling.cc:255] Waited 2.27 s on lock 0x3e4f970. stack: 00000000019065e1 0000000000976852 000000000078868f 0000000000a22fbf 0000000000a23be2 0000000001911c69 00007fd760afa181 00007fd75f9e247c ffffffffffffffff 1010 20:59:08.563463 (+ 3us) inbound_call.cc:130] Queueing success response Metrics: {"spinlock_wait_cycles":5229219712} I1010 20:59:08.563486 2748 raft_consensus.cc:1640] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 5 LEADER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 for term 6 because replica is either leader or believes a valid leader to be alive. I1010 20:59:08.563549 3451 raft_consensus.cc:825] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 6 FOLLOWER]: Rejecting Update request from peer f6d15ccd64ec49be8396ed78e2a88377 for earlier term 5. Current term is 6. Ops: [] I1010 20:59:08.563583 3389 leader_election.cc:361] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 6 election: Vote denied by peer f6d15ccd64ec49be8396ed78e2a88377. Message: Invalid argument: T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 5 LEADER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 for term 6 because replica is either leader or believes a valid leader to be alive. W1010 20:59:08.563616 3155 leader_election.cc:333] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 5 election: Vote denied by peer 42e94fc3adbb419591836d222d946978 with higher term. Message: Service unavailable: T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 for term 5 because replica is already servicing an update from a current leader or another vote. I1010 20:59:08.563628 3155 leader_election.cc:336] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term I1010 20:59:08.563633 3387 leader_election.cc:361] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 6 election: Vote denied by peer 1380a27d6b824575a3a5e139bee139f4. Message: Invalid argument: T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 for term 6 because replica is either leader or believes a valid leader to be alive. I1010 20:59:08.563647 3387 leader_election.cc:248] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. W1010 20:59:08.563683 3881 consensus_meta.cc:188] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978: Time spent flushing consensus metadata: real 0.812s user 0.000s sys 0.004s I1010 20:59:08.564002 3881 leader_election.cc:223] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 4 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:59:08.564028 4275 consensus_queue.cc:662] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [LEADER]: Peer responded invalid term: Peer: 42e94fc3adbb419591836d222d946978, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:59:08.564036 3881 leader_election.cc:223] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 4 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:59:08.564112 3451 raft_consensus.cc:1609] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 4: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. W1010 20:59:08.564111 3671 consensus_meta.cc:188] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93: Time spent flushing consensus metadata: real 0.795s user 0.000s sys 0.000s I1010 20:59:08.564132 4330 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.500s I1010 20:59:08.564134 2748 raft_consensus.cc:1609] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 4: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:59:08.564142 4330 raft_consensus.cc:1823] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 42e94fc3adbb419591836d222d946978 with higher term. Message: Service unavailable: T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 for term 5 because replica is already servicing an update from a current leader or another vote. I1010 20:59:08.564194 3389 leader_election.cc:361] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 4 election: Vote denied by peer f6d15ccd64ec49be8396ed78e2a88377. Message: Invalid argument: T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 in current term 4: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:59:08.564198 2684 leader_election.cc:361] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 4 election: Vote denied by peer 42e94fc3adbb419591836d222d946978. Message: Invalid argument: T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 4: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:59:08.564229 4331 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.537s I1010 20:59:08.564261 4331 raft_consensus.cc:1823] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: None given I1010 20:59:08.564260 3671 leader_election.cc:223] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 4 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:59:08.564280 2861 raft_consensus.cc:1654] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 for term 4 because replica is already servicing an update from a current leader or another vote. I1010 20:59:08.564335 4332 raft_consensus.cc:2026] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 5 LEADER]: Stepping down as leader of term 5 I1010 20:59:08.564350 4332 raft_consensus.cc:502] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f6d15ccd64ec49be8396ed78e2a88377, State: 1, Role: LEADER Watermarks: {Received: term: 5 index: 1 Committed: term: 5 index: 1} I1010 20:59:08.564371 4332 consensus_queue.cc:168] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.1, Current term: 5, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:59:08.564378 3390 leader_election.cc:361] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 4 election: Vote denied by peer 8de6cfa864d64c58bf4eafb689517c93. Message: Service unavailable: T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 for term 4 because replica is already servicing an update from a current leader or another vote. I1010 20:59:08.564390 3390 leader_election.cc:248] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. I1010 20:59:08.564450 4332 consensus_peers.cc:357] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 -> Peer 42e94fc3adbb419591836d222d946978 (127.6.150.8:35299): Closing peer: 42e94fc3adbb419591836d222d946978 I1010 20:59:08.564287 3671 leader_election.cc:223] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 4 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:59:08.564502 4332 consensus_peers.cc:357] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 -> Peer 1380a27d6b824575a3a5e139bee139f4 (127.6.150.6:46774): Closing peer: 1380a27d6b824575a3a5e139bee139f4 I1010 20:59:08.564539 4332 raft_consensus.cc:2031] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 5 FOLLOWER]: Advancing to term 6 I1010 20:59:08.564680 2862 raft_consensus.cc:1981] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.764s I1010 20:59:08.564893 2865 raft_consensus.cc:1609] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 4: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. I1010 20:59:08.565014 2685 leader_election.cc:361] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 4 election: Vote denied by peer 8de6cfa864d64c58bf4eafb689517c93. Message: Invalid argument: T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f6d15ccd64ec49be8396ed78e2a88377 in current term 4: Already voted for candidate 8de6cfa864d64c58bf4eafb689517c93 in this term. I1010 20:59:08.565033 2685 leader_election.cc:248] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. I1010 20:59:08.565048 3451 raft_consensus.cc:1609] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 4: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:59:08.565101 2748 raft_consensus.cc:1609] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 4: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:59:08.565198 2801 leader_election.cc:361] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 4 election: Vote denied by peer f6d15ccd64ec49be8396ed78e2a88377. Message: Invalid argument: T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 4: Already voted for candidate f6d15ccd64ec49be8396ed78e2a88377 in this term. I1010 20:59:08.565237 2801 leader_election.cc:361] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 4 election: Vote denied by peer 42e94fc3adbb419591836d222d946978. Message: Invalid argument: T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8de6cfa864d64c58bf4eafb689517c93 in current term 4: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:59:08.565246 2801 leader_election.cc:248] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. I1010 20:59:08.565304 4333 raft_consensus.cc:1981] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.797s I1010 20:59:08.565318 4333 raft_consensus.cc:1823] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: None given I1010 20:59:08.565351 4335 raft_consensus.cc:1981] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.065s I1010 20:59:08.565363 4334 raft_consensus.cc:1981] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.980s I1010 20:59:08.565366 4335 raft_consensus.cc:1823] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: None given I1010 20:59:08.565372 4334 raft_consensus.cc:1823] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: None given I1010 20:59:08.565933 3337 raft_consensus.cc:1677] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8062d66fd284035ae3bc2567bc11e7c in term 5. I1010 20:59:08.566047 2921 leader_election.cc:347] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 5 election: Vote granted by peer 7f623b00f5ae4b9c8e8940692d5dfd44 I1010 20:59:08.566072 2921 leader_election.cc:248] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. I1010 20:59:08.566112 4249 raft_consensus.cc:1981] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.383s I1010 20:59:08.566126 4249 raft_consensus.cc:1861] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 5 FOLLOWER]: Leader election won for term 5 I1010 20:59:08.566256 4249 raft_consensus.cc:467] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [term 5 LEADER]: Becoming Leader. State: Replica: b8062d66fd284035ae3bc2567bc11e7c, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:59:08.566290 4249 consensus_queue.cc:151] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 5, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:59:08.566737 2402 catalog_manager.cc:2018] T de9d164718e74d3db902ea603ab32407 reported consensus state change: term changed from 0 to 5, leader changed from to b8062d66fd284035ae3bc2567bc11e7c (127.6.150.4). New consensus state: current_term: 5 leader_uuid: "b8062d66fd284035ae3bc2567bc11e7c" config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } } I1010 20:59:08.568199 2862 raft_consensus.cc:1677] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8062d66fd284035ae3bc2567bc11e7c in term 5. I1010 20:59:08.568342 2921 leader_election.cc:347] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [CANDIDATE]: Term 5 election: Vote granted by peer 8de6cfa864d64c58bf4eafb689517c93 I1010 20:59:09.062263 3099 raft_consensus.cc:857] T 8da5904fd56d441ea29e61eb9d2b0b2f P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Refusing update from remote peer b8062d66fd284035ae3bc2567bc11e7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 4 index: 1. (index mismatch) I1010 20:59:09.062321 2508 raft_consensus.cc:857] T 8da5904fd56d441ea29e61eb9d2b0b2f P 84bde34331b94c4ea003e48f9317f792 [term 4 FOLLOWER]: Refusing update from remote peer b8062d66fd284035ae3bc2567bc11e7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 4 index: 1. (index mismatch) I1010 20:59:09.062412 4325 consensus_queue.cc:652] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [LEADER]: Connected to new peer: Peer: 33ef4317e50042ba95e517992caa6531, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:59:09.062599 4339 consensus_queue.cc:652] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c [LEADER]: Connected to new peer: Peer: 84bde34331b94c4ea003e48f9317f792, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:59:09.066771 2862 raft_consensus.cc:857] T de9d164718e74d3db902ea603ab32407 P 8de6cfa864d64c58bf4eafb689517c93 [term 5 FOLLOWER]: Refusing update from remote peer b8062d66fd284035ae3bc2567bc11e7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 5 index: 1. (index mismatch) I1010 20:59:09.066784 3337 raft_consensus.cc:857] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 5 FOLLOWER]: Refusing update from remote peer b8062d66fd284035ae3bc2567bc11e7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 5 index: 1. (index mismatch) I1010 20:59:09.066889 4249 consensus_queue.cc:652] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [LEADER]: Connected to new peer: Peer: 8de6cfa864d64c58bf4eafb689517c93, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:59:09.067028 4249 consensus_queue.cc:652] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c [LEADER]: Connected to new peer: Peer: 7f623b00f5ae4b9c8e8940692d5dfd44, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:59:09.456318 3842 raft_consensus.cc:375] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:09.456334 3842 raft_consensus.cc:2031] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 3 FOLLOWER]: Advancing to term 4 I1010 20:59:09.456724 3842 raft_consensus.cc:1981] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.809s I1010 20:59:09.456789 3842 raft_consensus.cc:391] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:59:09.457833 3842 leader_election.cc:223] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 4 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:59:09.457867 3842 leader_election.cc:223] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 4 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 I1010 20:59:09.458055 2748 raft_consensus.cc:2031] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 3 FOLLOWER]: Advancing to term 4 I1010 20:59:09.458082 3337 raft_consensus.cc:2031] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Advancing to term 4 I1010 20:59:09.458367 2748 raft_consensus.cc:1981] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.020s I1010 20:59:09.458469 3337 raft_consensus.cc:1981] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.041s I1010 20:59:09.459044 3337 raft_consensus.cc:1677] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33ef4317e50042ba95e517992caa6531 in term 4. I1010 20:59:09.459065 2748 raft_consensus.cc:1677] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33ef4317e50042ba95e517992caa6531 in term 4. I1010 20:59:09.459148 3039 leader_election.cc:347] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 4 election: Vote granted by peer 7f623b00f5ae4b9c8e8940692d5dfd44 I1010 20:59:09.459177 3039 leader_election.cc:248] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. I1010 20:59:09.459210 3038 leader_election.cc:347] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 4 election: Vote granted by peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:59:09.459354 4347 raft_consensus.cc:1981] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.566s I1010 20:59:09.459369 4347 raft_consensus.cc:1861] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Leader election won for term 4 I1010 20:59:09.459559 4347 raft_consensus.cc:467] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [term 4 LEADER]: Becoming Leader. State: Replica: 33ef4317e50042ba95e517992caa6531, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:59:09.459600 4347 consensus_queue.cc:151] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 4, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:59:09.460151 2402 catalog_manager.cc:2018] T 4c179cfa32ee4128ad57d34140e5320f reported consensus state change: term changed from 0 to 4, leader changed from to 33ef4317e50042ba95e517992caa6531 (127.6.150.5). New consensus state: current_term: 4 leader_uuid: "33ef4317e50042ba95e517992caa6531" config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } } I1010 20:59:09.824326 3722 raft_consensus.cc:375] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 4 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:09.824342 3722 raft_consensus.cc:2031] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 4 FOLLOWER]: Advancing to term 5 I1010 20:59:09.824592 3722 raft_consensus.cc:1981] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.380s I1010 20:59:09.824637 3722 raft_consensus.cc:391] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 5 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } I1010 20:59:09.825413 3722 leader_election.cc:223] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 5 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:59:09.825445 3722 leader_election.cc:223] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 5 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:59:09.825601 2748 raft_consensus.cc:2031] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Advancing to term 5 I1010 20:59:09.825609 2986 raft_consensus.cc:2031] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 4 FOLLOWER]: Advancing to term 5 I1010 20:59:09.826004 2748 raft_consensus.cc:1981] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.734s I1010 20:59:09.826022 2986 raft_consensus.cc:1981] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.794s I1010 20:59:09.826761 2986 raft_consensus.cc:1677] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 580cbaf7092444dfbc6fb344da88d7b6 in term 5. I1010 20:59:09.826854 3507 leader_election.cc:347] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 5 election: Vote granted by peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:59:09.826876 3507 leader_election.cc:248] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. I1010 20:59:09.827040 4351 raft_consensus.cc:1981] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.547s I1010 20:59:09.827066 4351 raft_consensus.cc:1861] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 5 FOLLOWER]: Leader election won for term 5 I1010 20:59:09.827200 2748 raft_consensus.cc:1677] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 580cbaf7092444dfbc6fb344da88d7b6 in term 5. I1010 20:59:09.827203 4351 raft_consensus.cc:467] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [term 5 LEADER]: Becoming Leader. State: Replica: 580cbaf7092444dfbc6fb344da88d7b6, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:59:09.827246 4351 consensus_queue.cc:151] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 5, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } I1010 20:59:09.827286 3506 leader_election.cc:347] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 5 election: Vote granted by peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:59:09.827988 2402 catalog_manager.cc:2018] T efe8fb09a1234c9fb1dc99948737a894 reported consensus state change: term changed from 0 to 5, leader changed from to 580cbaf7092444dfbc6fb344da88d7b6 (127.6.150.9). New consensus state: current_term: 5 leader_uuid: "580cbaf7092444dfbc6fb344da88d7b6" config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } } I1010 20:59:09.960001 2748 raft_consensus.cc:857] T 4c179cfa32ee4128ad57d34140e5320f P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Refusing update from remote peer 33ef4317e50042ba95e517992caa6531: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 4 index: 1. (index mismatch) I1010 20:59:09.960062 3337 raft_consensus.cc:857] T 4c179cfa32ee4128ad57d34140e5320f P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 4 FOLLOWER]: Refusing update from remote peer 33ef4317e50042ba95e517992caa6531: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 4 index: 1. (index mismatch) I1010 20:59:09.960093 4347 consensus_queue.cc:652] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [LEADER]: Connected to new peer: Peer: f6d15ccd64ec49be8396ed78e2a88377, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:59:09.960278 4347 consensus_queue.cc:652] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 [LEADER]: Connected to new peer: Peer: 7f623b00f5ae4b9c8e8940692d5dfd44, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:59:10.327697 2986 raft_consensus.cc:857] T efe8fb09a1234c9fb1dc99948737a894 P b8062d66fd284035ae3bc2567bc11e7c [term 5 FOLLOWER]: Refusing update from remote peer 580cbaf7092444dfbc6fb344da88d7b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 5 index: 1. (index mismatch) I1010 20:59:10.327776 2748 raft_consensus.cc:857] T efe8fb09a1234c9fb1dc99948737a894 P f6d15ccd64ec49be8396ed78e2a88377 [term 5 FOLLOWER]: Refusing update from remote peer 580cbaf7092444dfbc6fb344da88d7b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 5 index: 1. (index mismatch) I1010 20:59:10.327819 4351 consensus_queue.cc:652] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [LEADER]: Connected to new peer: Peer: b8062d66fd284035ae3bc2567bc11e7c, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:59:10.327968 4351 consensus_queue.cc:652] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 [LEADER]: Connected to new peer: Peer: f6d15ccd64ec49be8396ed78e2a88377, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:59:10.479666 3763 raft_consensus.cc:375] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 6 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:10.479683 3763 raft_consensus.cc:2031] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 6 FOLLOWER]: Advancing to term 7 I1010 20:59:10.480058 3763 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 7 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.715s I1010 20:59:10.480105 3763 raft_consensus.cc:391] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 7 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } I1010 20:59:10.480900 3763 leader_election.cc:223] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 7 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 I1010 20:59:10.480964 3763 leader_election.cc:223] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 7 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:59:10.481137 3220 raft_consensus.cc:2031] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 5 FOLLOWER]: Advancing to term 7 I1010 20:59:10.481194 3451 raft_consensus.cc:2031] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 6 FOLLOWER]: Advancing to term 7 I1010 20:59:10.481515 3451 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 7 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.813s I1010 20:59:10.481530 3220 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 7 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.560s I1010 20:59:10.482197 3220 raft_consensus.cc:1677] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d15ccd64ec49be8396ed78e2a88377 in term 7. I1010 20:59:10.482223 3451 raft_consensus.cc:1677] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d15ccd64ec49be8396ed78e2a88377 in term 7. I1010 20:59:10.482329 2682 leader_election.cc:347] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 7 election: Vote granted by peer 1380a27d6b824575a3a5e139bee139f4 I1010 20:59:10.482341 2682 leader_election.cc:248] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. I1010 20:59:10.482393 2684 leader_election.cc:347] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [CANDIDATE]: Term 7 election: Vote granted by peer 42e94fc3adbb419591836d222d946978 I1010 20:59:10.482523 4363 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 7 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.566s I1010 20:59:10.482542 4363 raft_consensus.cc:1861] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 7 FOLLOWER]: Leader election won for term 7 I1010 20:59:10.482745 4363 raft_consensus.cc:467] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [term 7 LEADER]: Becoming Leader. State: Replica: f6d15ccd64ec49be8396ed78e2a88377, State: 1, Role: LEADER Watermarks: {Received: term: 5 index: 1 Committed: term: 5 index: 1} I1010 20:59:10.482795 4363 consensus_queue.cc:151] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.1, Current term: 7, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } I1010 20:59:10.483377 2402 catalog_manager.cc:2018] T 90fa7ae3643e4575a6be1c4936b6cc11 reported consensus state change: term changed from 5 to 7. New consensus state: current_term: 7 leader_uuid: "f6d15ccd64ec49be8396ed78e2a88377" config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } } I1010 20:59:10.641121 3781 raft_consensus.cc:375] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 4 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:10.641154 3781 raft_consensus.cc:2031] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 4 FOLLOWER]: Advancing to term 5 I1010 20:59:10.641518 3781 raft_consensus.cc:1981] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.550s I1010 20:59:10.641561 3781 raft_consensus.cc:391] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 5 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:59:10.642349 3781 leader_election.cc:223] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 5 election: Requesting vote from peer 33ef4317e50042ba95e517992caa6531 I1010 20:59:10.642418 3781 leader_election.cc:223] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 5 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 I1010 20:59:10.642550 3099 raft_consensus.cc:2031] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Advancing to term 5 I1010 20:59:10.642622 2508 raft_consensus.cc:2031] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 4 FOLLOWER]: Advancing to term 5 I1010 20:59:10.642873 3099 raft_consensus.cc:1981] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.821s I1010 20:59:10.642972 2508 raft_consensus.cc:1981] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.016s I1010 20:59:10.643488 3099 raft_consensus.cc:1677] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de6cfa864d64c58bf4eafb689517c93 in term 5. I1010 20:59:10.643502 2508 raft_consensus.cc:1677] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de6cfa864d64c58bf4eafb689517c93 in term 5. I1010 20:59:10.643589 2799 leader_election.cc:347] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 5 election: Vote granted by peer 33ef4317e50042ba95e517992caa6531 I1010 20:59:10.643610 2799 leader_election.cc:248] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. I1010 20:59:10.643620 2802 leader_election.cc:347] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [CANDIDATE]: Term 5 election: Vote granted by peer 84bde34331b94c4ea003e48f9317f792 I1010 20:59:10.643756 4367 raft_consensus.cc:1981] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.349s I1010 20:59:10.643774 4367 raft_consensus.cc:1861] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 5 FOLLOWER]: Leader election won for term 5 I1010 20:59:10.643930 4367 raft_consensus.cc:467] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [term 5 LEADER]: Becoming Leader. State: Replica: 8de6cfa864d64c58bf4eafb689517c93, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:59:10.644023 4367 consensus_queue.cc:151] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 5, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:59:10.644556 2402 catalog_manager.cc:2018] T 19f492d94e864f1ebd1c70810b0470f7 reported consensus state change: term changed from 0 to 5, leader changed from to 8de6cfa864d64c58bf4eafb689517c93 (127.6.150.3). New consensus state: current_term: 5 leader_uuid: "8de6cfa864d64c58bf4eafb689517c93" config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } } I1010 20:59:10.983259 3220 raft_consensus.cc:857] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 7 FOLLOWER]: Refusing update from remote peer f6d15ccd64ec49be8396ed78e2a88377: Log matching property violated. Preceding OpId in replica: term: 5 index: 1. Preceding OpId from leader: term: 7 index: 2. (index mismatch) I1010 20:59:10.983331 3451 raft_consensus.cc:857] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 7 FOLLOWER]: Refusing update from remote peer f6d15ccd64ec49be8396ed78e2a88377: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 7 index: 2. (index mismatch) I1010 20:59:10.983376 4363 consensus_queue.cc:652] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [LEADER]: Connected to new peer: Peer: 1380a27d6b824575a3a5e139bee139f4, Is new: false, Last received: 5.1, Next index: 2, Last known committed idx: 1, Last exchange result: ERROR, Needs tablet copy: false I1010 20:59:10.983563 4363 consensus_queue.cc:652] T 90fa7ae3643e4575a6be1c4936b6cc11 P f6d15ccd64ec49be8396ed78e2a88377 [LEADER]: Connected to new peer: Peer: 42e94fc3adbb419591836d222d946978, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:59:11.144457 3099 raft_consensus.cc:857] T 19f492d94e864f1ebd1c70810b0470f7 P 33ef4317e50042ba95e517992caa6531 [term 5 FOLLOWER]: Refusing update from remote peer 8de6cfa864d64c58bf4eafb689517c93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 5 index: 1. (index mismatch) I1010 20:59:11.144531 2508 raft_consensus.cc:857] T 19f492d94e864f1ebd1c70810b0470f7 P 84bde34331b94c4ea003e48f9317f792 [term 5 FOLLOWER]: Refusing update from remote peer 8de6cfa864d64c58bf4eafb689517c93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 5 index: 1. (index mismatch) I1010 20:59:11.144587 4367 consensus_queue.cc:652] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [LEADER]: Connected to new peer: Peer: 33ef4317e50042ba95e517992caa6531, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:59:11.144712 4367 consensus_queue.cc:652] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 [LEADER]: Connected to new peer: Peer: 84bde34331b94c4ea003e48f9317f792, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false /data/10/dinesh/1.0.1/apache-kudu-1.0.1/src/kudu/integration-tests/create-table-itest.cc:138: Failure Failed Bad status: Timed out: Timed out waiting for Table Creation I1010 20:59:11.777585 1686 external_mini_cluster-itest-base.h:43] Found fatal failure I1010 20:59:11.777601 1686 external_mini_cluster-itest-base.h:49] Attempting to dump stacks of TS 0 with UUID 84bde34331b94c4ea003e48f9317f792 and pid 2443 ************************ BEGIN STACKS ************************** I1010 20:59:11.883447 3713 raft_consensus.cc:375] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:11.883492 3713 raft_consensus.cc:2031] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 4 FOLLOWER]: Advancing to term 5 I1010 20:59:11.883955 3713 raft_consensus.cc:1981] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.779s I1010 20:59:11.884006 3713 raft_consensus.cc:391] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 5 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } I1010 20:59:11.885082 3713 leader_election.cc:223] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 5 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:59:11.885113 3713 leader_election.cc:223] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 5 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 I1010 20:59:11.885242 2628 raft_consensus.cc:2031] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 4 FOLLOWER]: Advancing to term 5 I1010 20:59:11.885269 2748 raft_consensus.cc:2031] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Advancing to term 5 I1010 20:59:11.885578 2628 raft_consensus.cc:1981] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.654s I1010 20:59:11.885684 2748 raft_consensus.cc:1981] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.884s I1010 20:59:11.886152 2628 raft_consensus.cc:1677] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33ef4317e50042ba95e517992caa6531 in term 5. I1010 20:59:11.886257 3037 leader_election.cc:347] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 5 election: Vote granted by peer 1031512f5120422e9e22a0afabd78f61 I1010 20:59:11.886271 3037 leader_election.cc:248] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. I1010 20:59:11.886277 2748 raft_consensus.cc:1677] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33ef4317e50042ba95e517992caa6531 in term 5. I1010 20:59:11.886360 3038 leader_election.cc:347] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [CANDIDATE]: Term 5 election: Vote granted by peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:59:11.886445 4382 raft_consensus.cc:1981] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.303s I1010 20:59:11.886466 4382 raft_consensus.cc:1861] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 5 FOLLOWER]: Leader election won for term 5 I1010 20:59:11.886706 4382 raft_consensus.cc:467] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [term 5 LEADER]: Becoming Leader. State: Replica: 33ef4317e50042ba95e517992caa6531, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:59:11.886791 4382 consensus_queue.cc:151] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 5, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } I1010 20:59:11.887384 2406 catalog_manager.cc:2018] T 05a5666441ec49d792222a3e8674edf7 reported consensus state change: term changed from 0 to 5, leader changed from to 33ef4317e50042ba95e517992caa6531 (127.6.150.5). New consensus state: current_term: 5 leader_uuid: "33ef4317e50042ba95e517992caa6531" config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ef4317e50042ba95e517992caa6531" member_type: VOTER last_known_addr { host: "127.6.150.5" port: 40764 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } } W1010 20:59:11.906469 3622 catalog_manager.cc:2975] Tablet 872042177fee42c9b6c380a1b657755e (table test-table [id=6095d493aa4d412a8ffc0874a373d7f6]) was not created within the allowed timeout. Replacing with a new tablet 2f7a8f9befa745c2969913bbbe3e6b0c I1010 20:59:11.906850 3622 catalog_manager.cc:2864] Sending DeleteTablet for 3 replicas of tablet 872042177fee42c9b6c380a1b657755e I1010 20:59:11.906868 3622 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 872042177fee42c9b6c380a1b657755e on 8de6cfa864d64c58bf4eafb689517c93 (127.6.150.3:40117) (Replaced by 2f7a8f9befa745c2969913bbbe3e6b0c at 2016-10-10 20:59:11 UTC) I1010 20:59:11.906882 3622 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 872042177fee42c9b6c380a1b657755e on f6d15ccd64ec49be8396ed78e2a88377 (127.6.150.2:33554) (Replaced by 2f7a8f9befa745c2969913bbbe3e6b0c at 2016-10-10 20:59:11 UTC) I1010 20:59:11.906898 3622 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 872042177fee42c9b6c380a1b657755e on 42e94fc3adbb419591836d222d946978 (127.6.150.8:35299) (Replaced by 2f7a8f9befa745c2969913bbbe3e6b0c at 2016-10-10 20:59:11 UTC) I1010 20:59:11.907023 2843 tablet_service.cc:660] Processing DeleteTablet for tablet 872042177fee42c9b6c380a1b657755e with delete_type TABLET_DATA_DELETED (Replaced by 2f7a8f9befa745c2969913bbbe3e6b0c at 2016-10-10 20:59:11 UTC) from {real_user=slave, eff_user=} at 127.0.0.1:35643 I1010 20:59:11.907066 2843 tablet_peer.cc:215] Initiating TabletPeer shutdown for tablet: 872042177fee42c9b6c380a1b657755e I1010 20:59:11.907073 2843 maintenance_manager.cc:182] Unregistered op CompactRowSetsOp(872042177fee42c9b6c380a1b657755e) I1010 20:59:11.907076 2843 maintenance_manager.cc:182] Unregistered op MinorDeltaCompactionOp(872042177fee42c9b6c380a1b657755e) I1010 20:59:11.907080 2843 maintenance_manager.cc:182] Unregistered op MajorDeltaCompactionOp(872042177fee42c9b6c380a1b657755e) I1010 20:59:11.907085 2843 maintenance_manager.cc:182] Unregistered op FlushMRSOp(872042177fee42c9b6c380a1b657755e) I1010 20:59:11.907080 2607 tablet_service.cc:619] Processing CreateTablet for tablet 2f7a8f9befa745c2969913bbbe3e6b0c (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (0), range: [, ) I1010 20:59:11.907089 2843 maintenance_manager.cc:182] Unregistered op FlushDeltaMemStoresOp(872042177fee42c9b6c380a1b657755e) I1010 20:59:11.907093 2843 maintenance_manager.cc:182] Unregistered op LogGCOp(872042177fee42c9b6c380a1b657755e) I1010 20:59:11.907104 2843 raft_consensus.cc:1495] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 4 FOLLOWER]: Raft consensus shutting down. I1010 20:59:11.907112 2843 raft_consensus.cc:1511] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93 [term 4 FOLLOWER]: Raft consensus is shut down! I1010 20:59:11.907217 3430 tablet_service.cc:660] Processing DeleteTablet for tablet 872042177fee42c9b6c380a1b657755e with delete_type TABLET_DATA_DELETED (Replaced by 2f7a8f9befa745c2969913bbbe3e6b0c at 2016-10-10 20:59:11 UTC) from {real_user=slave, eff_user=} at 127.0.0.1:37365 I1010 20:59:11.907223 2724 tablet_service.cc:660] Processing DeleteTablet for tablet 872042177fee42c9b6c380a1b657755e with delete_type TABLET_DATA_DELETED (Replaced by 2f7a8f9befa745c2969913bbbe3e6b0c at 2016-10-10 20:59:11 UTC) from {real_user=slave, eff_user=} at 127.0.0.1:49508 I1010 20:59:11.907239 3430 tablet_peer.cc:215] Initiating TabletPeer shutdown for tablet: 872042177fee42c9b6c380a1b657755e I1010 20:59:11.907244 2724 tablet_peer.cc:215] Initiating TabletPeer shutdown for tablet: 872042177fee42c9b6c380a1b657755e I1010 20:59:11.907248 3430 maintenance_manager.cc:182] Unregistered op CompactRowSetsOp(872042177fee42c9b6c380a1b657755e) I1010 20:59:11.907253 2724 maintenance_manager.cc:182] Unregistered op CompactRowSetsOp(872042177fee42c9b6c380a1b657755e) I1010 20:59:11.907253 3430 maintenance_manager.cc:182] Unregistered op MinorDeltaCompactionOp(872042177fee42c9b6c380a1b657755e) I1010 20:59:11.907258 2724 maintenance_manager.cc:182] Unregistered op MinorDeltaCompactionOp(872042177fee42c9b6c380a1b657755e) I1010 20:59:11.907263 2724 maintenance_manager.cc:182] Unregistered op MajorDeltaCompactionOp(872042177fee42c9b6c380a1b657755e) I1010 20:59:11.907263 3430 maintenance_manager.cc:182] Unregistered op MajorDeltaCompactionOp(872042177fee42c9b6c380a1b657755e) I1010 20:59:11.907269 2724 maintenance_manager.cc:182] Unregistered op FlushMRSOp(872042177fee42c9b6c380a1b657755e) I1010 20:59:11.907270 3430 maintenance_manager.cc:182] Unregistered op FlushMRSOp(872042177fee42c9b6c380a1b657755e) I1010 20:59:11.907272 2724 maintenance_manager.cc:182] Unregistered op FlushDeltaMemStoresOp(872042177fee42c9b6c380a1b657755e) I1010 20:59:11.907274 3430 maintenance_manager.cc:182] Unregistered op FlushDeltaMemStoresOp(872042177fee42c9b6c380a1b657755e) I1010 20:59:11.907277 2724 maintenance_manager.cc:182] Unregistered op LogGCOp(872042177fee42c9b6c380a1b657755e) I1010 20:59:11.907280 3430 maintenance_manager.cc:182] Unregistered op LogGCOp(872042177fee42c9b6c380a1b657755e) I1010 20:59:11.907285 2724 raft_consensus.cc:1495] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Raft consensus shutting down. I1010 20:59:11.907290 3430 raft_consensus.cc:1495] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 4 FOLLOWER]: Raft consensus shutting down. I1010 20:59:11.907291 2724 raft_consensus.cc:1511] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377 [term 4 FOLLOWER]: Raft consensus is shut down! I1010 20:59:11.907296 3430 raft_consensus.cc:1511] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978 [term 4 FOLLOWER]: Raft consensus is shut down! I1010 20:59:11.907424 2843 ts_tablet_manager.cc:916] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93: Deleting tablet data with delete state TABLET_DATA_DELETED I1010 20:59:11.907502 3314 tablet_service.cc:619] Processing CreateTablet for tablet 2f7a8f9befa745c2969913bbbe3e6b0c (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (0), range: [, ) I1010 20:59:11.907572 2607 ts_tablet_manager.cc:766] Registered tablet 2f7a8f9befa745c2969913bbbe3e6b0c I1010 20:59:11.907964 2724 ts_tablet_manager.cc:916] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377: Deleting tablet data with delete state TABLET_DATA_DELETED I1010 20:59:11.907974 3314 ts_tablet_manager.cc:766] Registered tablet 2f7a8f9befa745c2969913bbbe3e6b0c I1010 20:59:11.908171 3549 tablet_service.cc:619] Processing CreateTablet for tablet 2f7a8f9befa745c2969913bbbe3e6b0c (table=test-table [id=6095d493aa4d412a8ffc0874a373d7f6]), partition=hash buckets: (0), range: [, ) I1010 20:59:11.908563 2843 ts_tablet_manager.cc:926] T 872042177fee42c9b6c380a1b657755e P 8de6cfa864d64c58bf4eafb689517c93: Tablet deleted. Last logged OpId: 0.0 I1010 20:59:11.908674 4389 ts_tablet_manager.cc:632] T 2f7a8f9befa745c2969913bbbe3e6b0c P 1031512f5120422e9e22a0afabd78f61: Bootstrapping tablet I1010 20:59:11.908808 4389 tablet_bootstrap.cc:380] T 2f7a8f9befa745c2969913bbbe3e6b0c P 1031512f5120422e9e22a0afabd78f61: Bootstrap starting. I1010 20:59:11.908902 3549 ts_tablet_manager.cc:766] Registered tablet 2f7a8f9befa745c2969913bbbe3e6b0c I1010 20:59:11.909018 4389 tablet_bootstrap.cc:542] T 2f7a8f9befa745c2969913bbbe3e6b0c P 1031512f5120422e9e22a0afabd78f61: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:59:11.909137 4389 tablet_bootstrap.cc:482] T 2f7a8f9befa745c2969913bbbe3e6b0c P 1031512f5120422e9e22a0afabd78f61: No blocks or log segments found. Creating new log. I1010 20:59:11.909137 2395 catalog_manager.cc:2567] TS 8de6cfa864d64c58bf4eafb689517c93 (127.6.150.3:40117): tablet 872042177fee42c9b6c380a1b657755e (table test-table [id=6095d493aa4d412a8ffc0874a373d7f6]) successfully deleted I1010 20:59:11.909333 3430 ts_tablet_manager.cc:916] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978: Deleting tablet data with delete state TABLET_DATA_DELETED I1010 20:59:11.909389 2724 ts_tablet_manager.cc:926] T 872042177fee42c9b6c380a1b657755e P f6d15ccd64ec49be8396ed78e2a88377: Tablet deleted. Last logged OpId: 0.0 I1010 20:59:11.909721 2394 catalog_manager.cc:2567] TS f6d15ccd64ec49be8396ed78e2a88377 (127.6.150.2:33554): tablet 872042177fee42c9b6c380a1b657755e (table test-table [id=6095d493aa4d412a8ffc0874a373d7f6]) successfully deleted I1010 20:59:11.910025 4391 ts_tablet_manager.cc:632] T 2f7a8f9befa745c2969913bbbe3e6b0c P 580cbaf7092444dfbc6fb344da88d7b6: Bootstrapping tablet I1010 20:59:11.910153 4390 ts_tablet_manager.cc:632] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44: Bootstrapping tablet I1010 20:59:11.910207 3430 ts_tablet_manager.cc:926] T 872042177fee42c9b6c380a1b657755e P 42e94fc3adbb419591836d222d946978: Tablet deleted. Last logged OpId: 0.0 I1010 20:59:11.910264 4391 tablet_bootstrap.cc:380] T 2f7a8f9befa745c2969913bbbe3e6b0c P 580cbaf7092444dfbc6fb344da88d7b6: Bootstrap starting. I1010 20:59:11.910353 4390 tablet_bootstrap.cc:380] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44: Bootstrap starting. I1010 20:59:11.910461 4391 tablet_bootstrap.cc:542] T 2f7a8f9befa745c2969913bbbe3e6b0c P 580cbaf7092444dfbc6fb344da88d7b6: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:59:11.910536 4390 tablet_bootstrap.cc:542] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:59:11.910544 4391 tablet_bootstrap.cc:482] T 2f7a8f9befa745c2969913bbbe3e6b0c P 580cbaf7092444dfbc6fb344da88d7b6: No blocks or log segments found. Creating new log. I1010 20:59:11.910660 4390 tablet_bootstrap.cc:482] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44: No blocks or log segments found. Creating new log. I1010 20:59:11.910687 2394 catalog_manager.cc:2567] TS 42e94fc3adbb419591836d222d946978 (127.6.150.8:35299): tablet 872042177fee42c9b6c380a1b657755e (table test-table [id=6095d493aa4d412a8ffc0874a373d7f6]) successfully deleted I1010 20:59:11.911140 4389 tablet_bootstrap.cc:380] T 2f7a8f9befa745c2969913bbbe3e6b0c P 1031512f5120422e9e22a0afabd78f61: No bootstrap required, opened a new log I1010 20:59:11.911161 4389 ts_tablet_manager.cc:637] T 2f7a8f9befa745c2969913bbbe3e6b0c P 1031512f5120422e9e22a0afabd78f61: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s I1010 20:59:11.911358 4391 tablet_bootstrap.cc:380] T 2f7a8f9befa745c2969913bbbe3e6b0c P 580cbaf7092444dfbc6fb344da88d7b6: No bootstrap required, opened a new log I1010 20:59:11.911378 4391 ts_tablet_manager.cc:637] T 2f7a8f9befa745c2969913bbbe3e6b0c P 580cbaf7092444dfbc6fb344da88d7b6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1010 20:59:11.911569 4389 raft_consensus.cc:275] T 2f7a8f9befa745c2969913bbbe3e6b0c P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:59:11.911604 4389 raft_consensus.cc:306] T 2f7a8f9befa745c2969913bbbe3e6b0c P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:59:11.911617 4389 raft_consensus.cc:502] T 2f7a8f9befa745c2969913bbbe3e6b0c P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1031512f5120422e9e22a0afabd78f61, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:59:11.911625 4389 consensus_queue.cc:168] T 2f7a8f9befa745c2969913bbbe3e6b0c P 1031512f5120422e9e22a0afabd78f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:59:11.911691 4391 raft_consensus.cc:275] T 2f7a8f9befa745c2969913bbbe3e6b0c P 580cbaf7092444dfbc6fb344da88d7b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:59:11.911722 4391 raft_consensus.cc:306] T 2f7a8f9befa745c2969913bbbe3e6b0c P 580cbaf7092444dfbc6fb344da88d7b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:59:11.911734 4391 raft_consensus.cc:502] T 2f7a8f9befa745c2969913bbbe3e6b0c P 580cbaf7092444dfbc6fb344da88d7b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 580cbaf7092444dfbc6fb344da88d7b6, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:59:11.911741 4391 consensus_queue.cc:168] T 2f7a8f9befa745c2969913bbbe3e6b0c P 580cbaf7092444dfbc6fb344da88d7b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:59:11.912374 4389 ts_tablet_manager.cc:664] T 2f7a8f9befa745c2969913bbbe3e6b0c P 1031512f5120422e9e22a0afabd78f61: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1010 20:59:11.912595 4391 ts_tablet_manager.cc:664] T 2f7a8f9befa745c2969913bbbe3e6b0c P 580cbaf7092444dfbc6fb344da88d7b6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1010 20:59:11.912703 4390 tablet_bootstrap.cc:380] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44: No bootstrap required, opened a new log I1010 20:59:11.912719 4390 ts_tablet_manager.cc:637] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s I1010 20:59:11.913206 4390 raft_consensus.cc:275] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:59:11.913220 4390 raft_consensus.cc:306] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:59:11.913233 4390 raft_consensus.cc:502] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f623b00f5ae4b9c8e8940692d5dfd44, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:59:11.913239 4390 consensus_queue.cc:168] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:59:11.913511 4390 ts_tablet_manager.cc:664] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1010 20:59:12.197203 4402 raft_consensus.cc:375] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:12.197237 4402 raft_consensus.cc:2031] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:59:12.197679 4402 raft_consensus.cc:1981] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.634s I1010 20:59:12.197728 4402 raft_consensus.cc:391] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:59:12.198483 4402 leader_election.cc:223] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: Requesting vote from peer 580cbaf7092444dfbc6fb344da88d7b6 I1010 20:59:12.198523 4402 leader_election.cc:223] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 I1010 20:59:12.198640 3569 raft_consensus.cc:2031] T 2f7a8f9befa745c2969913bbbe3e6b0c P 580cbaf7092444dfbc6fb344da88d7b6 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:59:12.198719 2628 raft_consensus.cc:2031] T 2f7a8f9befa745c2969913bbbe3e6b0c P 1031512f5120422e9e22a0afabd78f61 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:59:12.198959 3569 raft_consensus.cc:1981] T 2f7a8f9befa745c2969913bbbe3e6b0c P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.612s I1010 20:59:12.199165 2628 raft_consensus.cc:1981] T 2f7a8f9befa745c2969913bbbe3e6b0c P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.587s I1010 20:59:12.199551 3569 raft_consensus.cc:1677] T 2f7a8f9befa745c2969913bbbe3e6b0c P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in term 1. I1010 20:59:12.199626 2628 raft_consensus.cc:1677] T 2f7a8f9befa745c2969913bbbe3e6b0c P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f623b00f5ae4b9c8e8940692d5dfd44 in term 1. I1010 20:59:12.199657 3273 leader_election.cc:347] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: Vote granted by peer 580cbaf7092444dfbc6fb344da88d7b6 I1010 20:59:12.199672 3273 leader_election.cc:248] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I1010 20:59:12.199717 4403 raft_consensus.cc:1981] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.554s I1010 20:59:12.199784 4403 raft_consensus.cc:1861] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 FOLLOWER]: Leader election won for term 1 I1010 20:59:12.199733 3271 leader_election.cc:347] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 1 election: Vote granted by peer 1031512f5120422e9e22a0afabd78f61 I1010 20:59:12.200011 4403 raft_consensus.cc:467] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 1 LEADER]: Becoming Leader. State: Replica: 7f623b00f5ae4b9c8e8940692d5dfd44, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:59:12.200074 4403 consensus_queue.cc:151] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:59:12.200688 2406 catalog_manager.cc:2018] T 2f7a8f9befa745c2969913bbbe3e6b0c reported consensus state change: term changed from 0 to 1, leader changed from to 7f623b00f5ae4b9c8e8940692d5dfd44 (127.6.150.7). New consensus state: current_term: 1 leader_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } } I1010 20:59:12.387173 2748 raft_consensus.cc:857] T 05a5666441ec49d792222a3e8674edf7 P f6d15ccd64ec49be8396ed78e2a88377 [term 5 FOLLOWER]: Refusing update from remote peer 33ef4317e50042ba95e517992caa6531: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 5 index: 1. (index mismatch) I1010 20:59:12.387264 2628 raft_consensus.cc:857] T 05a5666441ec49d792222a3e8674edf7 P 1031512f5120422e9e22a0afabd78f61 [term 5 FOLLOWER]: Refusing update from remote peer 33ef4317e50042ba95e517992caa6531: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 5 index: 1. (index mismatch) I1010 20:59:12.387298 4382 consensus_queue.cc:652] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [LEADER]: Connected to new peer: Peer: f6d15ccd64ec49be8396ed78e2a88377, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:59:12.387351 4382 consensus_queue.cc:652] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 [LEADER]: Connected to new peer: Peer: 1031512f5120422e9e22a0afabd78f61, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false Cannot access memory at address 0x480186b1a9358d50 Cannot access memory at address 0x480186b1a9358d48 /tmp/run_tha_testXiwMYW/build/release/test-work/create-table-itest/2443: No such file or directory. No threads. ************************* END STACKS *************************** I1010 20:59:12.499503 1686 external_mini_cluster-itest-base.h:49] Attempting to dump stacks of TS 1 with UUID 1031512f5120422e9e22a0afabd78f61 and pid 2561 ************************ BEGIN STACKS ************************** W1010 20:59:12.518087 2802 consensus_peers.cc:332] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 -> Peer 84bde34331b94c4ea003e48f9317f792 (127.6.150.0:40113): Couldn't send request to peer 84bde34331b94c4ea003e48f9317f792 for tablet 19f492d94e864f1ebd1c70810b0470f7. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.0:40113: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W1010 20:59:12.630127 2921 consensus_peers.cc:332] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c -> Peer 84bde34331b94c4ea003e48f9317f792 (127.6.150.0:40113): Couldn't send request to peer 84bde34331b94c4ea003e48f9317f792 for tablet 970154c84d6849cc83af1d9e5ad6980d. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.0:40113: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W1010 20:59:12.645323 2802 consensus_peers.cc:332] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 -> Peer 84bde34331b94c4ea003e48f9317f792 (127.6.150.0:40113): Couldn't send request to peer 84bde34331b94c4ea003e48f9317f792 for tablet 4aa163baddb74c4998157964dd55b1f4. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.0:40113: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. I1010 20:59:12.700600 3569 raft_consensus.cc:857] T 2f7a8f9befa745c2969913bbbe3e6b0c P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Refusing update from remote peer 7f623b00f5ae4b9c8e8940692d5dfd44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I1010 20:59:12.700721 2628 raft_consensus.cc:857] T 2f7a8f9befa745c2969913bbbe3e6b0c P 1031512f5120422e9e22a0afabd78f61 [term 1 FOLLOWER]: Refusing update from remote peer 7f623b00f5ae4b9c8e8940692d5dfd44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I1010 20:59:12.700767 4403 consensus_queue.cc:652] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44 [LEADER]: Connected to new peer: Peer: 580cbaf7092444dfbc6fb344da88d7b6, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I1010 20:59:12.700898 4403 consensus_queue.cc:652] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44 [LEADER]: Connected to new peer: Peer: 1031512f5120422e9e22a0afabd78f61, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false W1010 20:59:12.762509 3039 consensus_peers.cc:332] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 -> Peer 84bde34331b94c4ea003e48f9317f792 (127.6.150.0:40113): Couldn't send request to peer 84bde34331b94c4ea003e48f9317f792 for tablet 7d680e6840854bca9daef77d506408e9. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.0:40113: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W1010 20:59:12.833482 2921 consensus_peers.cc:332] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c -> Peer 84bde34331b94c4ea003e48f9317f792 (127.6.150.0:40113): Couldn't send request to peer 84bde34331b94c4ea003e48f9317f792 for tablet 8da5904fd56d441ea29e61eb9d2b0b2f. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.0:40113: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W1010 20:59:12.953320 2568 consensus_peers.cc:332] T 95e7c143b0524a9b9ade2cfcc081e679 P 1031512f5120422e9e22a0afabd78f61 -> Peer 84bde34331b94c4ea003e48f9317f792 (127.6.150.0:40113): Couldn't send request to peer 84bde34331b94c4ea003e48f9317f792 for tablet 95e7c143b0524a9b9ade2cfcc081e679. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.0:40113: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W1010 20:59:13.018069 2802 consensus_peers.cc:332] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 -> Peer 84bde34331b94c4ea003e48f9317f792 (127.6.150.0:40113): Couldn't send request to peer 84bde34331b94c4ea003e48f9317f792 for tablet 19f492d94e864f1ebd1c70810b0470f7. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.0:40113: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W1010 20:59:13.130110 2921 consensus_peers.cc:332] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c -> Peer 84bde34331b94c4ea003e48f9317f792 (127.6.150.0:40113): Couldn't send request to peer 84bde34331b94c4ea003e48f9317f792 for tablet 970154c84d6849cc83af1d9e5ad6980d. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.0:40113: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W1010 20:59:13.145316 2802 consensus_peers.cc:332] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 -> Peer 84bde34331b94c4ea003e48f9317f792 (127.6.150.0:40113): Couldn't send request to peer 84bde34331b94c4ea003e48f9317f792 for tablet 4aa163baddb74c4998157964dd55b1f4. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.0:40113: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. Cannot access memory at address 0x480186b1a9358d50 Cannot access memory at address 0x480186b1a9358d48 /tmp/run_tha_testXiwMYW/build/release/test-work/create-table-itest/2561: No such file or directory. No threads. ************************* END STACKS *************************** I1010 20:59:13.194010 1686 external_mini_cluster-itest-base.h:49] Attempting to dump stacks of TS 2 with UUID f6d15ccd64ec49be8396ed78e2a88377 and pid 2678 ************************ BEGIN STACKS ************************** W1010 20:59:13.262459 3039 consensus_peers.cc:332] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 -> Peer 84bde34331b94c4ea003e48f9317f792 (127.6.150.0:40113): Couldn't send request to peer 84bde34331b94c4ea003e48f9317f792 for tablet 7d680e6840854bca9daef77d506408e9. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.0:40113: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W1010 20:59:13.307512 3037 consensus_peers.cc:332] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 05a5666441ec49d792222a3e8674edf7. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W1010 20:59:13.333505 2921 consensus_peers.cc:332] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c -> Peer 84bde34331b94c4ea003e48f9317f792 (127.6.150.0:40113): Couldn't send request to peer 84bde34331b94c4ea003e48f9317f792 for tablet 8da5904fd56d441ea29e61eb9d2b0b2f. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.0:40113: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W1010 20:59:13.341962 2800 consensus_peers.cc:332] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 4aa163baddb74c4998157964dd55b1f4. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W1010 20:59:13.365917 3154 consensus_peers.cc:332] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 981834f5e88f4585b6420fca7ddcb289. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W1010 20:59:13.518167 2802 consensus_peers.cc:332] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 -> Peer 84bde34331b94c4ea003e48f9317f792 (127.6.150.0:40113): Couldn't send request to peer 84bde34331b94c4ea003e48f9317f792 for tablet 19f492d94e864f1ebd1c70810b0470f7. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.0:40113: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W1010 20:59:13.574137 3271 consensus_peers.cc:332] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 2f7a8f9befa745c2969913bbbe3e6b0c. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W1010 20:59:13.618096 3505 consensus_peers.cc:332] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 17a4ac4e4e164cebb4d88f59511cf926. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W1010 20:59:13.630101 2921 consensus_peers.cc:332] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c -> Peer 84bde34331b94c4ea003e48f9317f792 (127.6.150.0:40113): Couldn't send request to peer 84bde34331b94c4ea003e48f9317f792 for tablet 970154c84d6849cc83af1d9e5ad6980d. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.0:40113: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W1010 20:59:13.637881 2919 consensus_peers.cc:332] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 84fcfff9fdef4acf8c2b47f36330567f. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W1010 20:59:13.645380 2802 consensus_peers.cc:332] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 -> Peer 84bde34331b94c4ea003e48f9317f792 (127.6.150.0:40113): Couldn't send request to peer 84bde34331b94c4ea003e48f9317f792 for tablet 4aa163baddb74c4998157964dd55b1f4. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.0:40113: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W1010 20:59:13.762501 3039 consensus_peers.cc:332] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 -> Peer 84bde34331b94c4ea003e48f9317f792 (127.6.150.0:40113): Couldn't send request to peer 84bde34331b94c4ea003e48f9317f792 for tablet 7d680e6840854bca9daef77d506408e9. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.0:40113: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W1010 20:59:13.807569 3037 consensus_peers.cc:332] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 05a5666441ec49d792222a3e8674edf7. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. Cannot access memory at address 0x480186b1a9358d50 Cannot access memory at address 0x480186b1a9358d48 /tmp/run_tha_testXiwMYW/build/release/test-work/create-table-itest/2678: No such file or directory. No threads. ************************* END STACKS *************************** I1010 20:59:13.827394 1686 external_mini_cluster-itest-base.h:49] Attempting to dump stacks of TS 3 with UUID 8de6cfa864d64c58bf4eafb689517c93 and pid 2795 ************************ BEGIN STACKS ************************** W1010 20:59:13.833648 2921 consensus_peers.cc:332] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c -> Peer 84bde34331b94c4ea003e48f9317f792 (127.6.150.0:40113): Couldn't send request to peer 84bde34331b94c4ea003e48f9317f792 for tablet 8da5904fd56d441ea29e61eb9d2b0b2f. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.0:40113: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W1010 20:59:13.842167 2800 consensus_peers.cc:332] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 4aa163baddb74c4998157964dd55b1f4. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W1010 20:59:13.866000 3154 consensus_peers.cc:332] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 981834f5e88f4585b6420fca7ddcb289. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W1010 20:59:13.949805 3038 consensus_peers.cc:332] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 -> Peer f6d15ccd64ec49be8396ed78e2a88377 (127.6.150.2:33554): Couldn't send request to peer f6d15ccd64ec49be8396ed78e2a88377 for tablet 4c179cfa32ee4128ad57d34140e5320f. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.2:33554: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W1010 20:59:14.018406 2802 consensus_peers.cc:332] T 19f492d94e864f1ebd1c70810b0470f7 P 8de6cfa864d64c58bf4eafb689517c93 -> Peer 84bde34331b94c4ea003e48f9317f792 (127.6.150.0:40113): Couldn't send request to peer 84bde34331b94c4ea003e48f9317f792 for tablet 19f492d94e864f1ebd1c70810b0470f7. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.0:40113: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W1010 20:59:14.074193 3271 consensus_peers.cc:332] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 2f7a8f9befa745c2969913bbbe3e6b0c. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W1010 20:59:14.118418 3505 consensus_peers.cc:332] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 17a4ac4e4e164cebb4d88f59511cf926. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W1010 20:59:14.130301 2921 consensus_peers.cc:332] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c -> Peer 84bde34331b94c4ea003e48f9317f792 (127.6.150.0:40113): Couldn't send request to peer 84bde34331b94c4ea003e48f9317f792 for tablet 970154c84d6849cc83af1d9e5ad6980d. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.0:40113: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W1010 20:59:14.138037 2919 consensus_peers.cc:332] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 84fcfff9fdef4acf8c2b47f36330567f. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W1010 20:59:14.143831 3038 consensus_peers.cc:332] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 -> Peer f6d15ccd64ec49be8396ed78e2a88377 (127.6.150.2:33554): Couldn't send request to peer f6d15ccd64ec49be8396ed78e2a88377 for tablet 05a5666441ec49d792222a3e8674edf7. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.2:33554: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W1010 20:59:14.145457 2802 consensus_peers.cc:332] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 -> Peer 84bde34331b94c4ea003e48f9317f792 (127.6.150.0:40113): Couldn't send request to peer 84bde34331b94c4ea003e48f9317f792 for tablet 4aa163baddb74c4998157964dd55b1f4. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.0:40113: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W1010 20:59:14.240308 3389 consensus_peers.cc:332] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 -> Peer f6d15ccd64ec49be8396ed78e2a88377 (127.6.150.2:33554): Couldn't send request to peer f6d15ccd64ec49be8396ed78e2a88377 for tablet f689db1eddb54b1f85b7df38d14534b4. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.2:33554: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W1010 20:59:14.248672 3506 consensus_peers.cc:332] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer f6d15ccd64ec49be8396ed78e2a88377 (127.6.150.2:33554): Couldn't send request to peer f6d15ccd64ec49be8396ed78e2a88377 for tablet efe8fb09a1234c9fb1dc99948737a894. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.2:33554: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W1010 20:59:14.262620 3039 consensus_peers.cc:332] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 -> Peer 84bde34331b94c4ea003e48f9317f792 (127.6.150.0:40113): Couldn't send request to peer 84bde34331b94c4ea003e48f9317f792 for tablet 7d680e6840854bca9daef77d506408e9. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.0:40113: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W1010 20:59:14.307649 3037 consensus_peers.cc:332] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 05a5666441ec49d792222a3e8674edf7. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W1010 20:59:14.335500 2921 consensus_peers.cc:332] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c -> Peer 84bde34331b94c4ea003e48f9317f792 (127.6.150.0:40113): Couldn't send request to peer 84bde34331b94c4ea003e48f9317f792 for tablet 8da5904fd56d441ea29e61eb9d2b0b2f. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.0:40113: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W1010 20:59:14.342200 2800 consensus_peers.cc:332] T 4aa163baddb74c4998157964dd55b1f4 P 8de6cfa864d64c58bf4eafb689517c93 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 4aa163baddb74c4998157964dd55b1f4. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W1010 20:59:14.366094 3154 consensus_peers.cc:332] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 981834f5e88f4585b6420fca7ddcb289. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W1010 20:59:14.449813 3038 consensus_peers.cc:332] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 -> Peer f6d15ccd64ec49be8396ed78e2a88377 (127.6.150.2:33554): Couldn't send request to peer f6d15ccd64ec49be8396ed78e2a88377 for tablet 4c179cfa32ee4128ad57d34140e5320f. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.2:33554: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. Cannot access memory at address 0x480186b1a9358d50 Cannot access memory at address 0x480186b1a9358d48 /tmp/run_tha_testXiwMYW/build/release/test-work/create-table-itest/2795: No such file or directory. No threads. ************************* END STACKS *************************** I1010 20:59:14.524510 1686 external_mini_cluster-itest-base.h:49] Attempting to dump stacks of TS 4 with UUID b8062d66fd284035ae3bc2567bc11e7c and pid 2912 ************************ BEGIN STACKS ************************** W1010 20:59:14.574281 3271 consensus_peers.cc:332] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 2f7a8f9befa745c2969913bbbe3e6b0c. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W1010 20:59:14.618350 3505 consensus_peers.cc:332] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 17a4ac4e4e164cebb4d88f59511cf926. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W1010 20:59:14.630385 2921 consensus_peers.cc:332] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c -> Peer 84bde34331b94c4ea003e48f9317f792 (127.6.150.0:40113): Couldn't send request to peer 84bde34331b94c4ea003e48f9317f792 for tablet 970154c84d6849cc83af1d9e5ad6980d. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.0:40113: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. W1010 20:59:14.638082 2919 consensus_peers.cc:332] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 84fcfff9fdef4acf8c2b47f36330567f. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W1010 20:59:14.644093 3038 consensus_peers.cc:332] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 -> Peer f6d15ccd64ec49be8396ed78e2a88377 (127.6.150.2:33554): Couldn't send request to peer f6d15ccd64ec49be8396ed78e2a88377 for tablet 05a5666441ec49d792222a3e8674edf7. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.2:33554: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. I1010 20:59:14.663480 3755 raft_consensus.cc:371] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Failure of leader 1031512f5120422e9e22a0afabd78f61 detected. Triggering leader election I1010 20:59:14.663494 3755 raft_consensus.cc:2031] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 3 FOLLOWER]: Advancing to term 4 I1010 20:59:14.663875 3755 raft_consensus.cc:1981] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.528s I1010 20:59:14.663910 3755 raft_consensus.cc:391] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 4 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } I1010 20:59:14.665046 3755 leader_election.cc:223] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 4 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 I1010 20:59:14.665067 3755 leader_election.cc:223] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 4 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 W1010 20:59:14.665220 3271 leader_election.cc:275] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 1031512f5120422e9e22a0afabd78f61: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111) W1010 20:59:14.665240 3273 leader_election.cc:275] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 84bde34331b94c4ea003e48f9317f792: Network error: Client connection negotiation failed: client connection to 127.6.150.0:40113: connect: Connection refused (error 111) I1010 20:59:14.665256 3273 leader_election.cc:248] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. I1010 20:59:14.665410 4448 raft_consensus.cc:1981] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.562s I1010 20:59:14.665426 4448 raft_consensus.cc:1823] T 95e7c143b0524a9b9ade2cfcc081e679 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: None given W1010 20:59:14.668025 3156 consensus_peers.cc:332] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 -> Peer 8de6cfa864d64c58bf4eafb689517c93 (127.6.150.3:40117): Couldn't send request to peer 8de6cfa864d64c58bf4eafb689517c93 for tablet 949b26dfeadc4de38868bf4ed994d184. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.3:40117: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W1010 20:59:14.740314 3389 consensus_peers.cc:332] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 -> Peer f6d15ccd64ec49be8396ed78e2a88377 (127.6.150.2:33554): Couldn't send request to peer f6d15ccd64ec49be8396ed78e2a88377 for tablet f689db1eddb54b1f85b7df38d14534b4. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.2:33554: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W1010 20:59:14.748744 3506 consensus_peers.cc:332] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer f6d15ccd64ec49be8396ed78e2a88377 (127.6.150.2:33554): Couldn't send request to peer f6d15ccd64ec49be8396ed78e2a88377 for tablet efe8fb09a1234c9fb1dc99948737a894. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.2:33554: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W1010 20:59:14.762739 3039 consensus_peers.cc:332] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 -> Peer 84bde34331b94c4ea003e48f9317f792 (127.6.150.0:40113): Couldn't send request to peer 84bde34331b94c4ea003e48f9317f792 for tablet 7d680e6840854bca9daef77d506408e9. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.0:40113: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. W1010 20:59:14.807901 3037 consensus_peers.cc:332] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 05a5666441ec49d792222a3e8674edf7. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W1010 20:59:14.833845 2921 consensus_peers.cc:332] T 8da5904fd56d441ea29e61eb9d2b0b2f P b8062d66fd284035ae3bc2567bc11e7c -> Peer 84bde34331b94c4ea003e48f9317f792 (127.6.150.0:40113): Couldn't send request to peer 84bde34331b94c4ea003e48f9317f792 for tablet 8da5904fd56d441ea29e61eb9d2b0b2f. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.0:40113: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. W1010 20:59:14.866186 3154 consensus_peers.cc:332] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 981834f5e88f4585b6420fca7ddcb289. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. I1010 20:59:14.905455 3779 raft_consensus.cc:371] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 7 FOLLOWER]: Failure of leader f6d15ccd64ec49be8396ed78e2a88377 detected. Triggering leader election I1010 20:59:14.905468 3779 raft_consensus.cc:2031] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 7 FOLLOWER]: Advancing to term 8 I1010 20:59:14.905843 3779 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 8 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.585s I1010 20:59:14.905889 3779 raft_consensus.cc:391] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 8 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } I1010 20:59:14.906786 3779 leader_election.cc:223] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 8 election: Requesting vote from peer 1380a27d6b824575a3a5e139bee139f4 I1010 20:59:14.906816 3779 leader_election.cc:223] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 8 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 I1010 20:59:14.907076 3220 raft_consensus.cc:1640] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 for term 8 because replica is either leader or believes a valid leader to be alive. W1010 20:59:14.907129 3389 leader_election.cc:275] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer f6d15ccd64ec49be8396ed78e2a88377: Network error: Client connection negotiation failed: client connection to 127.6.150.2:33554: connect: Connection refused (error 111) I1010 20:59:14.907186 3387 leader_election.cc:361] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 8 election: Vote denied by peer 1380a27d6b824575a3a5e139bee139f4. Message: Invalid argument: T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e94fc3adbb419591836d222d946978 for term 8 because replica is either leader or believes a valid leader to be alive. I1010 20:59:14.907208 3387 leader_election.cc:248] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. I1010 20:59:14.907402 4449 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 8 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.559s I1010 20:59:14.907433 4449 raft_consensus.cc:1823] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: None given W1010 20:59:14.909745 3156 consensus_peers.cc:332] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 -> Peer 8de6cfa864d64c58bf4eafb689517c93 (127.6.150.3:40117): Couldn't send request to peer 8de6cfa864d64c58bf4eafb689517c93 for tablet 981834f5e88f4585b6420fca7ddcb289. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.3:40117: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W1010 20:59:14.916337 2921 consensus_peers.cc:332] T de9d164718e74d3db902ea603ab32407 P b8062d66fd284035ae3bc2567bc11e7c -> Peer 8de6cfa864d64c58bf4eafb689517c93 (127.6.150.3:40117): Couldn't send request to peer 8de6cfa864d64c58bf4eafb689517c93 for tablet de9d164718e74d3db902ea603ab32407. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.3:40117: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W1010 20:59:14.949862 3038 consensus_peers.cc:332] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 -> Peer f6d15ccd64ec49be8396ed78e2a88377 (127.6.150.2:33554): Couldn't send request to peer f6d15ccd64ec49be8396ed78e2a88377 for tablet 4c179cfa32ee4128ad57d34140e5320f. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.2:33554: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W1010 20:59:14.954658 3390 consensus_peers.cc:332] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 -> Peer 8de6cfa864d64c58bf4eafb689517c93 (127.6.150.3:40117): Couldn't send request to peer 8de6cfa864d64c58bf4eafb689517c93 for tablet 7836528275cb45aab827361611bb3bbe. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.3:40117: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W1010 20:59:15.074426 3271 consensus_peers.cc:332] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 2f7a8f9befa745c2969913bbbe3e6b0c. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W1010 20:59:15.118408 3505 consensus_peers.cc:332] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 17a4ac4e4e164cebb4d88f59511cf926. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W1010 20:59:15.130463 2921 consensus_peers.cc:332] T 970154c84d6849cc83af1d9e5ad6980d P b8062d66fd284035ae3bc2567bc11e7c -> Peer 84bde34331b94c4ea003e48f9317f792 (127.6.150.0:40113): Couldn't send request to peer 84bde34331b94c4ea003e48f9317f792 for tablet 970154c84d6849cc83af1d9e5ad6980d. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.0:40113: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times. W1010 20:59:15.138173 2919 consensus_peers.cc:332] T 84fcfff9fdef4acf8c2b47f36330567f P b8062d66fd284035ae3bc2567bc11e7c -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 84fcfff9fdef4acf8c2b47f36330567f. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W1010 20:59:15.144152 3038 consensus_peers.cc:332] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 -> Peer f6d15ccd64ec49be8396ed78e2a88377 (127.6.150.2:33554): Couldn't send request to peer f6d15ccd64ec49be8396ed78e2a88377 for tablet 05a5666441ec49d792222a3e8674edf7. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.2:33554: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W1010 20:59:15.168087 3156 consensus_peers.cc:332] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 -> Peer 8de6cfa864d64c58bf4eafb689517c93 (127.6.150.3:40117): Couldn't send request to peer 8de6cfa864d64c58bf4eafb689517c93 for tablet 949b26dfeadc4de38868bf4ed994d184. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.3:40117: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. Cannot access memory at address 0x480186b1a9358d50 Cannot access memory at address 0x480186b1a9358d48 /tmp/run_tha_testXiwMYW/build/release/test-work/create-table-itest/2912: No such file or directory. No threads. ************************* END STACKS *************************** I1010 20:59:15.224241 1686 external_mini_cluster-itest-base.h:49] Attempting to dump stacks of TS 5 with UUID 33ef4317e50042ba95e517992caa6531 and pid 3032 ************************ BEGIN STACKS ************************** W1010 20:59:15.240473 3389 consensus_peers.cc:332] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 -> Peer f6d15ccd64ec49be8396ed78e2a88377 (127.6.150.2:33554): Couldn't send request to peer f6d15ccd64ec49be8396ed78e2a88377 for tablet f689db1eddb54b1f85b7df38d14534b4. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.2:33554: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W1010 20:59:15.248845 3506 consensus_peers.cc:332] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer f6d15ccd64ec49be8396ed78e2a88377 (127.6.150.2:33554): Couldn't send request to peer f6d15ccd64ec49be8396ed78e2a88377 for tablet efe8fb09a1234c9fb1dc99948737a894. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.2:33554: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W1010 20:59:15.262866 3039 consensus_peers.cc:332] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 -> Peer 84bde34331b94c4ea003e48f9317f792 (127.6.150.0:40113): Couldn't send request to peer 84bde34331b94c4ea003e48f9317f792 for tablet 7d680e6840854bca9daef77d506408e9. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.0:40113: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times. W1010 20:59:15.307886 3037 consensus_peers.cc:332] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 05a5666441ec49d792222a3e8674edf7. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. W1010 20:59:15.366268 3154 consensus_peers.cc:332] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 981834f5e88f4585b6420fca7ddcb289. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. W1010 20:59:15.370172 3156 consensus_peers.cc:332] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 -> Peer b8062d66fd284035ae3bc2567bc11e7c (127.6.150.4:44055): Couldn't send request to peer b8062d66fd284035ae3bc2567bc11e7c for tablet 949b26dfeadc4de38868bf4ed994d184. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.4:44055: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W1010 20:59:15.409883 3156 consensus_peers.cc:332] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 -> Peer 8de6cfa864d64c58bf4eafb689517c93 (127.6.150.3:40117): Couldn't send request to peer 8de6cfa864d64c58bf4eafb689517c93 for tablet 981834f5e88f4585b6420fca7ddcb289. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.3:40117: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W1010 20:59:15.450006 3038 consensus_peers.cc:332] T 4c179cfa32ee4128ad57d34140e5320f P 33ef4317e50042ba95e517992caa6531 -> Peer f6d15ccd64ec49be8396ed78e2a88377 (127.6.150.2:33554): Couldn't send request to peer f6d15ccd64ec49be8396ed78e2a88377 for tablet 4c179cfa32ee4128ad57d34140e5320f. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.2:33554: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W1010 20:59:15.454735 3390 consensus_peers.cc:332] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 -> Peer 8de6cfa864d64c58bf4eafb689517c93 (127.6.150.3:40117): Couldn't send request to peer 8de6cfa864d64c58bf4eafb689517c93 for tablet 7836528275cb45aab827361611bb3bbe. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.3:40117: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. I1010 20:59:15.503482 3848 raft_consensus.cc:371] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 7 FOLLOWER]: Failure of leader f6d15ccd64ec49be8396ed78e2a88377 detected. Triggering leader election I1010 20:59:15.503495 3848 raft_consensus.cc:2031] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 7 FOLLOWER]: Advancing to term 8 I1010 20:59:15.503890 3848 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 8 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.562s I1010 20:59:15.503928 3848 raft_consensus.cc:391] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 8 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1380a27d6b824575a3a5e139bee139f4" member_type: VOTER last_known_addr { host: "127.6.150.6" port: 46774 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } peers { permanent_uuid: "f6d15ccd64ec49be8396ed78e2a88377" member_type: VOTER last_known_addr { host: "127.6.150.2" port: 33554 } } I1010 20:59:15.504777 3848 leader_election.cc:223] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 8 election: Requesting vote from peer 42e94fc3adbb419591836d222d946978 I1010 20:59:15.504812 3848 leader_election.cc:223] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 8 election: Requesting vote from peer f6d15ccd64ec49be8396ed78e2a88377 W1010 20:59:15.505019 3155 leader_election.cc:275] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer f6d15ccd64ec49be8396ed78e2a88377: Network error: Client connection negotiation failed: client connection to 127.6.150.2:33554: connect: Connection refused (error 111) I1010 20:59:15.505022 3451 raft_consensus.cc:1609] T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 in current term 8: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:59:15.505131 3155 leader_election.cc:361] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 8 election: Vote denied by peer 42e94fc3adbb419591836d222d946978. Message: Invalid argument: T 90fa7ae3643e4575a6be1c4936b6cc11 P 42e94fc3adbb419591836d222d946978 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1380a27d6b824575a3a5e139bee139f4 in current term 8: Already voted for candidate 42e94fc3adbb419591836d222d946978 in this term. I1010 20:59:15.505153 3155 leader_election.cc:248] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. I1010 20:59:15.505295 4456 raft_consensus.cc:1981] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 8 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.552s I1010 20:59:15.505316 4456 raft_consensus.cc:1823] T 90fa7ae3643e4575a6be1c4936b6cc11 P 1380a27d6b824575a3a5e139bee139f4 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: None given W1010 20:59:15.574455 3271 consensus_peers.cc:332] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 2f7a8f9befa745c2969913bbbe3e6b0c. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. W1010 20:59:15.585202 3507 consensus_peers.cc:332] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer b8062d66fd284035ae3bc2567bc11e7c (127.6.150.4:44055): Couldn't send request to peer b8062d66fd284035ae3bc2567bc11e7c for tablet efe8fb09a1234c9fb1dc99948737a894. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.4:44055: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W1010 20:59:15.618418 3505 consensus_peers.cc:332] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 17a4ac4e4e164cebb4d88f59511cf926. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. W1010 20:59:15.644121 3038 consensus_peers.cc:332] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 -> Peer f6d15ccd64ec49be8396ed78e2a88377 (127.6.150.2:33554): Couldn't send request to peer f6d15ccd64ec49be8396ed78e2a88377 for tablet 05a5666441ec49d792222a3e8674edf7. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.2:33554: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W1010 20:59:15.668202 3156 consensus_peers.cc:332] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 -> Peer 8de6cfa864d64c58bf4eafb689517c93 (127.6.150.3:40117): Couldn't send request to peer 8de6cfa864d64c58bf4eafb689517c93 for tablet 949b26dfeadc4de38868bf4ed994d184. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.3:40117: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W1010 20:59:15.682351 3273 consensus_peers.cc:332] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 -> Peer b8062d66fd284035ae3bc2567bc11e7c (127.6.150.4:44055): Couldn't send request to peer b8062d66fd284035ae3bc2567bc11e7c for tablet f2289fb138ea4a5ab6c02803c4e9aa51. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.4:44055: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W1010 20:59:15.740598 3389 consensus_peers.cc:332] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 -> Peer f6d15ccd64ec49be8396ed78e2a88377 (127.6.150.2:33554): Couldn't send request to peer f6d15ccd64ec49be8396ed78e2a88377 for tablet f689db1eddb54b1f85b7df38d14534b4. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.2:33554: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W1010 20:59:15.748901 3506 consensus_peers.cc:332] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer f6d15ccd64ec49be8396ed78e2a88377 (127.6.150.2:33554): Couldn't send request to peer f6d15ccd64ec49be8396ed78e2a88377 for tablet efe8fb09a1234c9fb1dc99948737a894. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.2:33554: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W1010 20:59:15.762933 3039 consensus_peers.cc:332] T 7d680e6840854bca9daef77d506408e9 P 33ef4317e50042ba95e517992caa6531 -> Peer 84bde34331b94c4ea003e48f9317f792 (127.6.150.0:40113): Couldn't send request to peer 84bde34331b94c4ea003e48f9317f792 for tablet 7d680e6840854bca9daef77d506408e9. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.0:40113: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 7 times. W1010 20:59:15.807962 3037 consensus_peers.cc:332] T 05a5666441ec49d792222a3e8674edf7 P 33ef4317e50042ba95e517992caa6531 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 05a5666441ec49d792222a3e8674edf7. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times. Cannot access memory at address 0x480186b1a9358d50 Cannot access memory at address 0x480186b1a9358d48 /tmp/run_tha_testXiwMYW/build/release/test-work/create-table-itest/3032: No such file or directory. No threads. ************************* END STACKS *************************** I1010 20:59:15.865526 1686 external_mini_cluster-itest-base.h:49] Attempting to dump stacks of TS 6 with UUID 1380a27d6b824575a3a5e139bee139f4 and pid 3149 W1010 20:59:15.866363 3154 consensus_peers.cc:332] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 981834f5e88f4585b6420fca7ddcb289. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times. ************************ BEGIN STACKS ************************** W1010 20:59:15.870280 3156 consensus_peers.cc:332] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 -> Peer b8062d66fd284035ae3bc2567bc11e7c (127.6.150.4:44055): Couldn't send request to peer b8062d66fd284035ae3bc2567bc11e7c for tablet 949b26dfeadc4de38868bf4ed994d184. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.4:44055: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W1010 20:59:15.909967 3156 consensus_peers.cc:332] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 -> Peer 8de6cfa864d64c58bf4eafb689517c93 (127.6.150.3:40117): Couldn't send request to peer 8de6cfa864d64c58bf4eafb689517c93 for tablet 981834f5e88f4585b6420fca7ddcb289. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.3:40117: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W1010 20:59:15.954869 3390 consensus_peers.cc:332] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 -> Peer 8de6cfa864d64c58bf4eafb689517c93 (127.6.150.3:40117): Couldn't send request to peer 8de6cfa864d64c58bf4eafb689517c93 for tablet 7836528275cb45aab827361611bb3bbe. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.3:40117: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W1010 20:59:16.074759 3271 consensus_peers.cc:332] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 2f7a8f9befa745c2969913bbbe3e6b0c. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times. W1010 20:59:16.085444 3507 consensus_peers.cc:332] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer b8062d66fd284035ae3bc2567bc11e7c (127.6.150.4:44055): Couldn't send request to peer b8062d66fd284035ae3bc2567bc11e7c for tablet efe8fb09a1234c9fb1dc99948737a894. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.4:44055: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W1010 20:59:16.118530 3505 consensus_peers.cc:332] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 17a4ac4e4e164cebb4d88f59511cf926. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times. W1010 20:59:16.145965 3504 consensus_peers.cc:332] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer 33ef4317e50042ba95e517992caa6531 (127.6.150.5:40764): Couldn't send request to peer 33ef4317e50042ba95e517992caa6531 for tablet ed24ca4817744964991178277119b367. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.5:40764: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W1010 20:59:16.168395 3156 consensus_peers.cc:332] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 -> Peer 8de6cfa864d64c58bf4eafb689517c93 (127.6.150.3:40117): Couldn't send request to peer 8de6cfa864d64c58bf4eafb689517c93 for tablet 949b26dfeadc4de38868bf4ed994d184. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.3:40117: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W1010 20:59:16.182466 3273 consensus_peers.cc:332] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 -> Peer b8062d66fd284035ae3bc2567bc11e7c (127.6.150.4:44055): Couldn't send request to peer b8062d66fd284035ae3bc2567bc11e7c for tablet f2289fb138ea4a5ab6c02803c4e9aa51. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.4:44055: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W1010 20:59:16.240690 3389 consensus_peers.cc:332] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 -> Peer f6d15ccd64ec49be8396ed78e2a88377 (127.6.150.2:33554): Couldn't send request to peer f6d15ccd64ec49be8396ed78e2a88377 for tablet f689db1eddb54b1f85b7df38d14534b4. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.2:33554: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. W1010 20:59:16.249017 3506 consensus_peers.cc:332] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer f6d15ccd64ec49be8396ed78e2a88377 (127.6.150.2:33554): Couldn't send request to peer f6d15ccd64ec49be8396ed78e2a88377 for tablet efe8fb09a1234c9fb1dc99948737a894. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.2:33554: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. I1010 20:59:16.361501 3684 raft_consensus.cc:371] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Failure of leader b8062d66fd284035ae3bc2567bc11e7c detected. Triggering leader election I1010 20:59:16.361516 3684 raft_consensus.cc:2031] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Advancing to term 3 I1010 20:59:16.361871 3684 raft_consensus.cc:1981] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.640s I1010 20:59:16.361906 3684 raft_consensus.cc:391] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "84bde34331b94c4ea003e48f9317f792" member_type: VOTER last_known_addr { host: "127.6.150.0" port: 40113 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } I1010 20:59:16.362978 3684 leader_election.cc:223] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 3 election: Requesting vote from peer 84bde34331b94c4ea003e48f9317f792 I1010 20:59:16.363009 3684 leader_election.cc:223] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 3 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c W1010 20:59:16.363267 3507 leader_election.cc:275] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 84bde34331b94c4ea003e48f9317f792: Network error: Client connection negotiation failed: client connection to 127.6.150.0:40113: connect: Connection refused (error 111) W1010 20:59:16.363286 3507 leader_election.cc:275] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer b8062d66fd284035ae3bc2567bc11e7c: Network error: Client connection negotiation failed: client connection to 127.6.150.4:44055: connect: Connection refused (error 111) I1010 20:59:16.363289 3507 leader_election.cc:248] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I1010 20:59:16.363369 4464 raft_consensus.cc:1981] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.558s I1010 20:59:16.363390 4464 raft_consensus.cc:1823] T 970154c84d6849cc83af1d9e5ad6980d P 580cbaf7092444dfbc6fb344da88d7b6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given W1010 20:59:16.366524 3154 consensus_peers.cc:332] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 981834f5e88f4585b6420fca7ddcb289. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 7 times. W1010 20:59:16.370435 3156 consensus_peers.cc:332] T 949b26dfeadc4de38868bf4ed994d184 P 1380a27d6b824575a3a5e139bee139f4 -> Peer b8062d66fd284035ae3bc2567bc11e7c (127.6.150.4:44055): Couldn't send request to peer b8062d66fd284035ae3bc2567bc11e7c for tablet 949b26dfeadc4de38868bf4ed994d184. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.4:44055: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W1010 20:59:16.410079 3156 consensus_peers.cc:332] T 981834f5e88f4585b6420fca7ddcb289 P 1380a27d6b824575a3a5e139bee139f4 -> Peer 8de6cfa864d64c58bf4eafb689517c93 (127.6.150.3:40117): Couldn't send request to peer 8de6cfa864d64c58bf4eafb689517c93 for tablet 981834f5e88f4585b6420fca7ddcb289. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.3:40117: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W1010 20:59:16.454936 3390 consensus_peers.cc:332] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 -> Peer 8de6cfa864d64c58bf4eafb689517c93 (127.6.150.3:40117): Couldn't send request to peer 8de6cfa864d64c58bf4eafb689517c93 for tablet 7836528275cb45aab827361611bb3bbe. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.3:40117: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. Cannot access memory at address 0x480186b1a9358d50 Cannot access memory at address 0x480186b1a9358d48 /tmp/run_tha_testXiwMYW/build/release/test-work/create-table-itest/3149: No such file or directory. No threads. ************************* END STACKS *************************** I1010 20:59:16.531033 1686 external_mini_cluster-itest-base.h:49] Attempting to dump stacks of TS 7 with UUID 7f623b00f5ae4b9c8e8940692d5dfd44 and pid 3266 ************************ BEGIN STACKS ************************** W1010 20:59:16.574723 3271 consensus_peers.cc:332] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 2f7a8f9befa745c2969913bbbe3e6b0c. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 7 times. W1010 20:59:16.585414 3507 consensus_peers.cc:332] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer b8062d66fd284035ae3bc2567bc11e7c (127.6.150.4:44055): Couldn't send request to peer b8062d66fd284035ae3bc2567bc11e7c for tablet efe8fb09a1234c9fb1dc99948737a894. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.4:44055: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W1010 20:59:16.618660 3505 consensus_peers.cc:332] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 17a4ac4e4e164cebb4d88f59511cf926. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 7 times. I1010 20:59:16.621855 3816 raft_consensus.cc:371] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 4 FOLLOWER]: Failure of leader b8062d66fd284035ae3bc2567bc11e7c detected. Triggering leader election I1010 20:59:16.621879 3816 raft_consensus.cc:2031] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 4 FOLLOWER]: Advancing to term 5 I1010 20:59:16.622220 3816 raft_consensus.cc:1981] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.511s I1010 20:59:16.622284 3816 raft_consensus.cc:391] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 5 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "42e94fc3adbb419591836d222d946978" member_type: VOTER last_known_addr { host: "127.6.150.8" port: 35299 } } I1010 20:59:16.622992 3816 leader_election.cc:223] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 5 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:59:16.623013 3816 leader_election.cc:223] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 5 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 W1010 20:59:16.623388 3388 leader_election.cc:275] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 1031512f5120422e9e22a0afabd78f61: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111) W1010 20:59:16.623447 3390 leader_election.cc:275] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer b8062d66fd284035ae3bc2567bc11e7c: Network error: Client connection negotiation failed: client connection to 127.6.150.4:44055: connect: Connection refused (error 111) I1010 20:59:16.623459 3390 leader_election.cc:248] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. I1010 20:59:16.623580 4472 raft_consensus.cc:1981] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 5 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.506s I1010 20:59:16.623594 4472 raft_consensus.cc:1823] T 84fcfff9fdef4acf8c2b47f36330567f P 42e94fc3adbb419591836d222d946978 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: None given W1010 20:59:16.646070 3504 consensus_peers.cc:332] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer 33ef4317e50042ba95e517992caa6531 (127.6.150.5:40764): Couldn't send request to peer 33ef4317e50042ba95e517992caa6531 for tablet ed24ca4817744964991178277119b367. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.5:40764: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W1010 20:59:16.649685 3387 consensus_peers.cc:332] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 -> Peer 1380a27d6b824575a3a5e139bee139f4 (127.6.150.6:46774): Couldn't send request to peer 1380a27d6b824575a3a5e139bee139f4 for tablet f689db1eddb54b1f85b7df38d14534b4. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.6:46774: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W1010 20:59:16.682550 3273 consensus_peers.cc:332] T f2289fb138ea4a5ab6c02803c4e9aa51 P 7f623b00f5ae4b9c8e8940692d5dfd44 -> Peer b8062d66fd284035ae3bc2567bc11e7c (127.6.150.4:44055): Couldn't send request to peer b8062d66fd284035ae3bc2567bc11e7c for tablet f2289fb138ea4a5ab6c02803c4e9aa51. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.4:44055: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W1010 20:59:16.740718 3389 consensus_peers.cc:332] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 -> Peer f6d15ccd64ec49be8396ed78e2a88377 (127.6.150.2:33554): Couldn't send request to peer f6d15ccd64ec49be8396ed78e2a88377 for tablet f689db1eddb54b1f85b7df38d14534b4. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.2:33554: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times. W1010 20:59:16.749115 3506 consensus_peers.cc:332] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer f6d15ccd64ec49be8396ed78e2a88377 (127.6.150.2:33554): Couldn't send request to peer f6d15ccd64ec49be8396ed78e2a88377 for tablet efe8fb09a1234c9fb1dc99948737a894. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.2:33554: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times. I1010 20:59:16.929757 3744 raft_consensus.cc:371] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 5 FOLLOWER]: Failure of leader b8062d66fd284035ae3bc2567bc11e7c detected. Triggering leader election I1010 20:59:16.929780 3744 raft_consensus.cc:2031] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 5 FOLLOWER]: Advancing to term 6 I1010 20:59:16.930150 3744 raft_consensus.cc:1981] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.599s I1010 20:59:16.930188 3744 raft_consensus.cc:391] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 6 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } peers { permanent_uuid: "b8062d66fd284035ae3bc2567bc11e7c" member_type: VOTER last_known_addr { host: "127.6.150.4" port: 44055 } } peers { permanent_uuid: "8de6cfa864d64c58bf4eafb689517c93" member_type: VOTER last_known_addr { host: "127.6.150.3" port: 40117 } } I1010 20:59:16.931062 3744 leader_election.cc:223] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 6 election: Requesting vote from peer b8062d66fd284035ae3bc2567bc11e7c I1010 20:59:16.931141 3744 leader_election.cc:223] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 6 election: Requesting vote from peer 8de6cfa864d64c58bf4eafb689517c93 W1010 20:59:16.931438 3273 leader_election.cc:275] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer b8062d66fd284035ae3bc2567bc11e7c: Network error: Client connection negotiation failed: client connection to 127.6.150.4:44055: connect: Connection refused (error 111) W1010 20:59:16.931458 3273 leader_election.cc:275] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 8de6cfa864d64c58bf4eafb689517c93: Network error: Client connection negotiation failed: client connection to 127.6.150.3:40117: connect: Connection refused (error 111) I1010 20:59:16.931462 3273 leader_election.cc:248] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. I1010 20:59:16.931578 4473 raft_consensus.cc:1981] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 6 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.501s I1010 20:59:16.931607 4473 raft_consensus.cc:1823] T de9d164718e74d3db902ea603ab32407 P 7f623b00f5ae4b9c8e8940692d5dfd44 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: None given W1010 20:59:16.954991 3390 consensus_peers.cc:332] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 -> Peer 8de6cfa864d64c58bf4eafb689517c93 (127.6.150.3:40117): Couldn't send request to peer 8de6cfa864d64c58bf4eafb689517c93 for tablet 7836528275cb45aab827361611bb3bbe. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.3:40117: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. W1010 20:59:17.074822 3271 consensus_peers.cc:332] T 2f7a8f9befa745c2969913bbbe3e6b0c P 7f623b00f5ae4b9c8e8940692d5dfd44 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 2f7a8f9befa745c2969913bbbe3e6b0c. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 8 times. W1010 20:59:17.085584 3507 consensus_peers.cc:332] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer b8062d66fd284035ae3bc2567bc11e7c (127.6.150.4:44055): Couldn't send request to peer b8062d66fd284035ae3bc2567bc11e7c for tablet efe8fb09a1234c9fb1dc99948737a894. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.4:44055: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W1010 20:59:17.118803 3505 consensus_peers.cc:332] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 17a4ac4e4e164cebb4d88f59511cf926. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 8 times. W1010 20:59:17.146174 3504 consensus_peers.cc:332] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer 33ef4317e50042ba95e517992caa6531 (127.6.150.5:40764): Couldn't send request to peer 33ef4317e50042ba95e517992caa6531 for tablet ed24ca4817744964991178277119b367. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.5:40764: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W1010 20:59:17.149853 3387 consensus_peers.cc:332] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 -> Peer 1380a27d6b824575a3a5e139bee139f4 (127.6.150.6:46774): Couldn't send request to peer 1380a27d6b824575a3a5e139bee139f4 for tablet f689db1eddb54b1f85b7df38d14534b4. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.6:46774: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. Cannot access memory at address 0x480186b1a9358d50 Cannot access memory at address 0x480186b1a9358d48 /tmp/run_tha_testXiwMYW/build/release/test-work/create-table-itest/3266: No such file or directory. No threads. ************************* END STACKS *************************** I1010 20:59:17.181597 1686 external_mini_cluster-itest-base.h:49] Attempting to dump stacks of TS 8 with UUID 42e94fc3adbb419591836d222d946978 and pid 3383 ************************ BEGIN STACKS ************************** W1010 20:59:17.234838 3507 consensus_peers.cc:332] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer 7f623b00f5ae4b9c8e8940692d5dfd44 (127.6.150.7:45436): Couldn't send request to peer 7f623b00f5ae4b9c8e8940692d5dfd44 for tablet 17a4ac4e4e164cebb4d88f59511cf926. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.7:45436: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W1010 20:59:17.240878 3389 consensus_peers.cc:332] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 -> Peer f6d15ccd64ec49be8396ed78e2a88377 (127.6.150.2:33554): Couldn't send request to peer f6d15ccd64ec49be8396ed78e2a88377 for tablet f689db1eddb54b1f85b7df38d14534b4. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.2:33554: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 7 times. W1010 20:59:17.249248 3506 consensus_peers.cc:332] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer f6d15ccd64ec49be8396ed78e2a88377 (127.6.150.2:33554): Couldn't send request to peer f6d15ccd64ec49be8396ed78e2a88377 for tablet efe8fb09a1234c9fb1dc99948737a894. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.2:33554: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 7 times. W1010 20:59:17.455198 3390 consensus_peers.cc:332] T 7836528275cb45aab827361611bb3bbe P 42e94fc3adbb419591836d222d946978 -> Peer 8de6cfa864d64c58bf4eafb689517c93 (127.6.150.3:40117): Couldn't send request to peer 8de6cfa864d64c58bf4eafb689517c93 for tablet 7836528275cb45aab827361611bb3bbe. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.3:40117: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times. W1010 20:59:17.585595 3507 consensus_peers.cc:332] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer b8062d66fd284035ae3bc2567bc11e7c (127.6.150.4:44055): Couldn't send request to peer b8062d66fd284035ae3bc2567bc11e7c for tablet efe8fb09a1234c9fb1dc99948737a894. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.4:44055: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. W1010 20:59:17.618917 3505 consensus_peers.cc:332] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 17a4ac4e4e164cebb4d88f59511cf926. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 9 times. W1010 20:59:17.625568 3507 consensus_peers.cc:332] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer 7f623b00f5ae4b9c8e8940692d5dfd44 (127.6.150.7:45436): Couldn't send request to peer 7f623b00f5ae4b9c8e8940692d5dfd44 for tablet ed24ca4817744964991178277119b367. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.7:45436: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W1010 20:59:17.646306 3504 consensus_peers.cc:332] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer 33ef4317e50042ba95e517992caa6531 (127.6.150.5:40764): Couldn't send request to peer 33ef4317e50042ba95e517992caa6531 for tablet ed24ca4817744964991178277119b367. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.5:40764: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W1010 20:59:17.649953 3387 consensus_peers.cc:332] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 -> Peer 1380a27d6b824575a3a5e139bee139f4 (127.6.150.6:46774): Couldn't send request to peer 1380a27d6b824575a3a5e139bee139f4 for tablet f689db1eddb54b1f85b7df38d14534b4. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.6:46774: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W1010 20:59:17.734863 3507 consensus_peers.cc:332] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer 7f623b00f5ae4b9c8e8940692d5dfd44 (127.6.150.7:45436): Couldn't send request to peer 7f623b00f5ae4b9c8e8940692d5dfd44 for tablet 17a4ac4e4e164cebb4d88f59511cf926. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.7:45436: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W1010 20:59:17.740972 3389 consensus_peers.cc:332] T f689db1eddb54b1f85b7df38d14534b4 P 42e94fc3adbb419591836d222d946978 -> Peer f6d15ccd64ec49be8396ed78e2a88377 (127.6.150.2:33554): Couldn't send request to peer f6d15ccd64ec49be8396ed78e2a88377 for tablet f689db1eddb54b1f85b7df38d14534b4. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.2:33554: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 8 times. W1010 20:59:17.749397 3506 consensus_peers.cc:332] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer f6d15ccd64ec49be8396ed78e2a88377 (127.6.150.2:33554): Couldn't send request to peer f6d15ccd64ec49be8396ed78e2a88377 for tablet efe8fb09a1234c9fb1dc99948737a894. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.2:33554: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 8 times. Cannot access memory at address 0x480186b1a9358d50 Cannot access memory at address 0x480186b1a9358d48 /tmp/run_tha_testXiwMYW/build/release/test-work/create-table-itest/3383: No such file or directory. No threads. ************************* END STACKS *************************** I1010 20:59:17.819550 1686 external_mini_cluster-itest-base.h:49] Attempting to dump stacks of TS 9 with UUID 580cbaf7092444dfbc6fb344da88d7b6 and pid 3500 ************************ BEGIN STACKS ************************** W1010 20:59:18.085911 3507 consensus_peers.cc:332] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer b8062d66fd284035ae3bc2567bc11e7c (127.6.150.4:44055): Couldn't send request to peer b8062d66fd284035ae3bc2567bc11e7c for tablet efe8fb09a1234c9fb1dc99948737a894. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.4:44055: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times. W1010 20:59:18.119016 3505 consensus_peers.cc:332] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer 1031512f5120422e9e22a0afabd78f61 (127.6.150.1:46963): Couldn't send request to peer 1031512f5120422e9e22a0afabd78f61 for tablet 17a4ac4e4e164cebb4d88f59511cf926. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 10 times. W1010 20:59:18.125679 3507 consensus_peers.cc:332] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer 7f623b00f5ae4b9c8e8940692d5dfd44 (127.6.150.7:45436): Couldn't send request to peer 7f623b00f5ae4b9c8e8940692d5dfd44 for tablet ed24ca4817744964991178277119b367. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.7:45436: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W1010 20:59:18.146347 3504 consensus_peers.cc:332] T ed24ca4817744964991178277119b367 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer 33ef4317e50042ba95e517992caa6531 (127.6.150.5:40764): Couldn't send request to peer 33ef4317e50042ba95e517992caa6531 for tablet ed24ca4817744964991178277119b367. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.5:40764: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. W1010 20:59:18.234874 3507 consensus_peers.cc:332] T 17a4ac4e4e164cebb4d88f59511cf926 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer 7f623b00f5ae4b9c8e8940692d5dfd44 (127.6.150.7:45436): Couldn't send request to peer 7f623b00f5ae4b9c8e8940692d5dfd44 for tablet 17a4ac4e4e164cebb4d88f59511cf926. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.7:45436: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W1010 20:59:18.249565 3506 consensus_peers.cc:332] T efe8fb09a1234c9fb1dc99948737a894 P 580cbaf7092444dfbc6fb344da88d7b6 -> Peer f6d15ccd64ec49be8396ed78e2a88377 (127.6.150.2:33554): Couldn't send request to peer f6d15ccd64ec49be8396ed78e2a88377 for tablet efe8fb09a1234c9fb1dc99948737a894. Status: Network error: Client connection negotiation failed: client connection to 127.6.150.2:33554: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 9 times. I1010 20:59:18.402699 4398 raft_consensus.cc:371] T 2f7a8f9befa745c2969913bbbe3e6b0c P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Failure of leader 7f623b00f5ae4b9c8e8940692d5dfd44 detected. Triggering leader election I1010 20:59:18.402724 4398 raft_consensus.cc:2031] T 2f7a8f9befa745c2969913bbbe3e6b0c P 580cbaf7092444dfbc6fb344da88d7b6 [term 1 FOLLOWER]: Advancing to term 2 I1010 20:59:18.403103 4398 raft_consensus.cc:1981] T 2f7a8f9befa745c2969913bbbe3e6b0c P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.609s I1010 20:59:18.403131 4398 raft_consensus.cc:391] T 2f7a8f9befa745c2969913bbbe3e6b0c P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580cbaf7092444dfbc6fb344da88d7b6" member_type: VOTER last_known_addr { host: "127.6.150.9" port: 51743 } } peers { permanent_uuid: "1031512f5120422e9e22a0afabd78f61" member_type: VOTER last_known_addr { host: "127.6.150.1" port: 46963 } } peers { permanent_uuid: "7f623b00f5ae4b9c8e8940692d5dfd44" member_type: VOTER last_known_addr { host: "127.6.150.7" port: 45436 } } I1010 20:59:18.404182 4398 leader_election.cc:223] T 2f7a8f9befa745c2969913bbbe3e6b0c P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 2 election: Requesting vote from peer 1031512f5120422e9e22a0afabd78f61 I1010 20:59:18.404214 4398 leader_election.cc:223] T 2f7a8f9befa745c2969913bbbe3e6b0c P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 2 election: Requesting vote from peer 7f623b00f5ae4b9c8e8940692d5dfd44 W1010 20:59:18.404377 3505 leader_election.cc:275] T 2f7a8f9befa745c2969913bbbe3e6b0c P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1031512f5120422e9e22a0afabd78f61: Network error: Client connection negotiation failed: client connection to 127.6.150.1:46963: connect: Connection refused (error 111) W1010 20:59:18.404438 3507 leader_election.cc:275] T 2f7a8f9befa745c2969913bbbe3e6b0c P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7f623b00f5ae4b9c8e8940692d5dfd44: Network error: Client connection negotiation failed: client connection to 127.6.150.7:45436: connect: Connection refused (error 111) I1010 20:59:18.404456 3507 leader_election.cc:248] T 2f7a8f9befa745c2969913bbbe3e6b0c P 580cbaf7092444dfbc6fb344da88d7b6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I1010 20:59:18.404594 4486 raft_consensus.cc:1981] T 2f7a8f9befa745c2969913bbbe3e6b0c P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.567s I1010 20:59:18.404613 4486 raft_consensus.cc:1823] T 2f7a8f9befa745c2969913bbbe3e6b0c P 580cbaf7092444dfbc6fb344da88d7b6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: None given Cannot access memory at address 0x480186b1a9358d50 Cannot access memory at address 0x480186b1a9358d48 /tmp/run_tha_testXiwMYW/build/release/test-work/create-table-itest/3500: No such file or directory. No threads. ************************* END STACKS *************************** I1010 20:59:18.469269 1686 external_mini_cluster.cc:687] Killing /tmp/run_tha_testXiwMYW/build/release/bin/kudu-master with pid 2388 I1010 20:59:18.471617 1686 test_util.cc:78] ----------------------------------------------- I1010 20:59:18.471640 1686 test_util.cc:79] Had fatal failures, leaving test files at /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestSpreadReplicasEvenly.1476133114077700-1686 [ FAILED ] CreateTableITest.TestSpreadReplicasEvenly (41808 ms) [ RUN ] CreateTableITest.TestCreateTableWithDeadTServers I1010 20:59:18.472152 1686 external_mini_cluster.cc:586] Running /tmp/run_tha_testXiwMYW/build/release/bin/kudu-master kudu-master --fs_wal_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateTableWithDeadTServers.1476133114077700-1686/minicluster-data/master-0 --fs_data_dirs=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateTableWithDeadTServers.1476133114077700-1686/minicluster-data/master-0 --rpc_bind_addresses=127.0.0.1:0 --webserver_interface=localhost --webserver_port=0 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateTableWithDeadTServers.1476133114077700-1686/minicluster-data/master-0 --tablet_creation_timeout_ms=1000 --tserver_unresponsive_timeout_ms=5000 --server_dump_info_path=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateTableWithDeadTServers.1476133114077700-1686/minicluster-data/master-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 --unlock_experimental_flags --unlock_unsafe_flags WARNING: Logging before InitGoogleLogging() is written to STDERR W1010 20:59:18.476563 4491 flags.cc:294] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true I1010 20:59:18.477087 4491 mem_tracker.cc:140] MemTracker: hard memory limit is 11.777661 GB I1010 20:59:18.477099 4491 mem_tracker.cc:142] MemTracker: soft memory limit is 7.066597 GB I1010 20:59:18.478142 4491 master_main.cc:58] Initializing master server... I1010 20:59:18.478238 4491 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 990115 I1010 20:59:18.478921 4491 server_base.cc:167] Could not load existing FS layout: Not found: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateTableWithDeadTServers.1476133114077700-1686/minicluster-data/master-0/instance: No such file or directory (error 2) I1010 20:59:18.478934 4491 server_base.cc:168] Creating new FS layout I1010 20:59:18.485074 4491 fs_manager.cc:354] Generated new instance metadata in path /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateTableWithDeadTServers.1476133114077700-1686/minicluster-data/master-0/instance: uuid: "c7eb565e98d74df4b890199f2e895330" format_stamp: "Formatted at 2016-10-10 20:59:18 on dist-test-slave-dist-test-slave-ymsa" I1010 20:59:18.491117 4491 fs_manager.cc:243] Opened local filesystem: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateTableWithDeadTServers.1476133114077700-1686/minicluster-data/master-0 uuid: "c7eb565e98d74df4b890199f2e895330" format_stamp: "Formatted at 2016-10-10 20:59:18 on dist-test-slave-dist-test-slave-ymsa" I1010 20:59:18.493100 4491 master_main.cc:61] Starting Master server... I1010 20:59:18.496611 4491 rpc_server.cc:164] RPC server started. Bound to: 127.0.0.1:33283 I1010 20:59:18.496644 4491 webserver.cc:125] Starting webserver on localhost:0 I1010 20:59:18.496647 4491 webserver.cc:136] Document root disabled I1010 20:59:18.496814 4491 net_util.cc:195] Address 127.0.0.1:0 for localhost:0 duplicates an earlier resolved entry. I1010 20:59:18.496897 4491 webserver.cc:220] Webserver started. Bound to: http://127.0.0.1:53220/ I1010 20:59:18.496992 4491 server_base.cc:239] Dumped server information to /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateTableWithDeadTServers.1476133114077700-1686/minicluster-data/master-0/info.pb I1010 20:59:18.501930 4545 tablet_bootstrap.cc:380] T 00000000000000000000000000000000 P c7eb565e98d74df4b890199f2e895330: Bootstrap starting. I1010 20:59:18.502055 4545 tablet_bootstrap.cc:542] T 00000000000000000000000000000000 P c7eb565e98d74df4b890199f2e895330: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I1010 20:59:18.502159 4545 tablet_bootstrap.cc:482] T 00000000000000000000000000000000 P c7eb565e98d74df4b890199f2e895330: No blocks or log segments found. Creating new log. I1010 20:59:18.502249 4545 log.cc:347] Log is configured to *not* fsync() on all Append() calls I1010 20:59:18.502650 4545 tablet_bootstrap.cc:380] T 00000000000000000000000000000000 P c7eb565e98d74df4b890199f2e895330: No bootstrap required, opened a new log I1010 20:59:18.502923 4545 raft_consensus.cc:275] T 00000000000000000000000000000000 P c7eb565e98d74df4b890199f2e895330 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7eb565e98d74df4b890199f2e895330" member_type: VOTER } I1010 20:59:18.502941 4545 raft_consensus.cc:306] T 00000000000000000000000000000000 P c7eb565e98d74df4b890199f2e895330 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1010 20:59:18.502950 4545 raft_consensus.cc:502] T 00000000000000000000000000000000 P c7eb565e98d74df4b890199f2e895330 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7eb565e98d74df4b890199f2e895330, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:59:18.502957 4545 consensus_queue.cc:168] T 00000000000000000000000000000000 P c7eb565e98d74df4b890199f2e895330 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I1010 20:59:18.502961 4545 raft_consensus.cc:319] T 00000000000000000000000000000000 P c7eb565e98d74df4b890199f2e895330 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I1010 20:59:18.502969 1686 external_mini_cluster.cc:622] Started /tmp/run_tha_testXiwMYW/build/release/bin/kudu-master as pid 4491 I1010 20:59:18.502993 4545 raft_consensus.cc:375] T 00000000000000000000000000000000 P c7eb565e98d74df4b890199f2e895330 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I1010 20:59:18.503000 4545 raft_consensus.cc:2031] T 00000000000000000000000000000000 P c7eb565e98d74df4b890199f2e895330 [term 0 FOLLOWER]: Advancing to term 1 I1010 20:59:18.503036 1686 external_mini_cluster.cc:586] Running /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateTableWithDeadTServers.1476133114077700-1686/minicluster-data/ts-0 --fs_data_dirs=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateTableWithDeadTServers.1476133114077700-1686/minicluster-data/ts-0 --rpc_bind_addresses=127.6.150.0:0 --local_ip_for_outbound_sockets=127.6.150.0 --webserver_interface=127.6.150.0 --webserver_port=0 --tserver_master_addrs=127.0.0.1:33283 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateTableWithDeadTServers.1476133114077700-1686/minicluster-data/ts-0 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateTableWithDeadTServers.1476133114077700-1686/minicluster-data/ts-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 --unlock_experimental_flags --unlock_unsafe_flags I1010 20:59:18.503237 4545 raft_consensus.cc:1981] T 00000000000000000000000000000000 P c7eb565e98d74df4b890199f2e895330 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.531s I1010 20:59:18.503259 4545 raft_consensus.cc:391] T 00000000000000000000000000000000 P c7eb565e98d74df4b890199f2e895330 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7eb565e98d74df4b890199f2e895330" member_type: VOTER } I1010 20:59:18.503876 4545 leader_election.cc:248] T 00000000000000000000000000000000 P c7eb565e98d74df4b890199f2e895330 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. WARNING: Logging before InitGoogleLogging() is written to STDERR W1010 20:59:18.507845 4549 flags.cc:294] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1010 20:59:18.507910 4549 flags.cc:294] Enabled unsafe flag: --never_fsync=true W1010 20:59:18.508294 4549 flags.cc:294] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.150.0 I1010 20:59:18.508370 4549 mem_tracker.cc:140] MemTracker: hard memory limit is 11.777661 GB I1010 20:59:18.508380 4549 mem_tracker.cc:142] MemTracker: soft memory limit is 7.066597 GB I1010 20:59:18.509260 4550 raft_consensus.cc:1981] T 00000000000000000000000000000000 P c7eb565e98d74df4b890199f2e895330 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.596s I1010 20:59:18.509277 4550 raft_consensus.cc:1861] T 00000000000000000000000000000000 P c7eb565e98d74df4b890199f2e895330 [term 1 FOLLOWER]: Leader election won for term 1 I1010 20:59:18.509310 4549 tablet_server_main.cc:54] Initializing tablet server... I1010 20:59:18.509433 4549 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 990115 I1010 20:59:18.510059 4549 server_base.cc:167] Could not load existing FS layout: Not found: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateTableWithDeadTServers.1476133114077700-1686/minicluster-data/ts-0/instance: No such file or directory (error 2) I1010 20:59:18.510071 4549 server_base.cc:168] Creating new FS layout I1010 20:59:18.511114 4549 fs_manager.cc:354] Generated new instance metadata in path /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateTableWithDeadTServers.1476133114077700-1686/minicluster-data/ts-0/instance: uuid: "d28b59bf702d4784a663d8e6458ffe07" format_stamp: "Formatted at 2016-10-10 20:59:18 on dist-test-slave-dist-test-slave-ymsa" I1010 20:59:18.511678 4550 raft_consensus.cc:467] T 00000000000000000000000000000000 P c7eb565e98d74df4b890199f2e895330 [term 1 LEADER]: Becoming Leader. State: Replica: c7eb565e98d74df4b890199f2e895330, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I1010 20:59:18.511698 4550 consensus_queue.cc:151] T 00000000000000000000000000000000 P c7eb565e98d74df4b890199f2e895330 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 1, Majority size: 1, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7eb565e98d74df4b890199f2e895330" member_type: VOTER } I1010 20:59:18.511736 4549 fs_manager.cc:243] Opened local filesystem: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateTableWithDeadTServers.1476133114077700-1686/minicluster-data/ts-0 uuid: "d28b59bf702d4784a663d8e6458ffe07" format_stamp: "Formatted at 2016-10-10 20:59:18 on dist-test-slave-dist-test-slave-ymsa" I1010 20:59:18.512768 4553 sys_catalog.cc:257] T 00000000000000000000000000000000 P c7eb565e98d74df4b890199f2e895330 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7eb565e98d74df4b890199f2e895330" config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7eb565e98d74df4b890199f2e895330" member_type: VOTER } } I1010 20:59:18.512783 4553 sys_catalog.cc:260] T 00000000000000000000000000000000 P c7eb565e98d74df4b890199f2e895330 [sys.catalog]: This master's current role is: LEADER I1010 20:59:18.512897 4545 sys_catalog.cc:257] T 00000000000000000000000000000000 P c7eb565e98d74df4b890199f2e895330 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletPeer. Latest consensus state: current_term: 1 leader_uuid: "c7eb565e98d74df4b890199f2e895330" config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7eb565e98d74df4b890199f2e895330" member_type: VOTER } } I1010 20:59:18.512914 4545 sys_catalog.cc:260] T 00000000000000000000000000000000 P c7eb565e98d74df4b890199f2e895330 [sys.catalog]: This master's current role is: LEADER I1010 20:59:18.512945 4555 sys_catalog.cc:257] T 00000000000000000000000000000000 P c7eb565e98d74df4b890199f2e895330 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7eb565e98d74df4b890199f2e895330. Latest consensus state: current_term: 1 leader_uuid: "c7eb565e98d74df4b890199f2e895330" config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7eb565e98d74df4b890199f2e895330" member_type: VOTER } } I1010 20:59:18.512969 4545 sys_catalog.cc:335] T 00000000000000000000000000000000 P c7eb565e98d74df4b890199f2e895330 [sys.catalog]: configured and running, proceeding with master startup. I1010 20:59:18.512977 4555 sys_catalog.cc:260] T 00000000000000000000000000000000 P c7eb565e98d74df4b890199f2e895330 [sys.catalog]: This master's current role is: LEADER W1010 20:59:18.513038 4562 catalog_manager.cc:365] Catalog manager background task thread going to sleep: Service unavailable: Catalog manager is not initialized. State: 1 I1010 20:59:18.513057 4491 master_main.cc:64] Master server successfully started. I1010 20:59:18.513057 4558 catalog_manager.cc:660] Loading table and tablet metadata into memory... I1010 20:59:18.513835 4549 tablet_server_main.cc:57] Starting tablet server... I1010 20:59:18.522300 4549 rpc_server.cc:164] RPC server started. Bound to: 127.6.150.0:48341 I1010 20:59:18.522367 4549 webserver.cc:125] Starting webserver on 127.6.150.0:0 I1010 20:59:18.522375 4549 webserver.cc:136] Document root disabled I1010 20:59:18.522467 4549 webserver.cc:220] Webserver started. Bound to: http://127.6.150.0:32964/ I1010 20:59:18.522572 4549 server_base.cc:239] Dumped server information to /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateTableWithDeadTServers.1476133114077700-1686/minicluster-data/ts-0/info.pb I1010 20:59:18.523591 1686 external_mini_cluster.cc:622] Started /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver as pid 4549 I1010 20:59:18.523655 1686 external_mini_cluster.cc:586] Running /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateTableWithDeadTServers.1476133114077700-1686/minicluster-data/ts-1 --fs_data_dirs=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateTableWithDeadTServers.1476133114077700-1686/minicluster-data/ts-1 --rpc_bind_addresses=127.6.150.1:0 --local_ip_for_outbound_sockets=127.6.150.1 --webserver_interface=127.6.150.1 --webserver_port=0 --tserver_master_addrs=127.0.0.1:33283 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateTableWithDeadTServers.1476133114077700-1686/minicluster-data/ts-1 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateTableWithDeadTServers.1476133114077700-1686/minicluster-data/ts-1/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 --unlock_experimental_flags --unlock_unsafe_flags WARNING: Logging before InitGoogleLogging() is written to STDERR W1010 20:59:18.527717 4671 flags.cc:294] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1010 20:59:18.527778 4671 flags.cc:294] Enabled unsafe flag: --never_fsync=true W1010 20:59:18.528134 4671 flags.cc:294] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.150.1 I1010 20:59:18.528208 4671 mem_tracker.cc:140] MemTracker: hard memory limit is 11.777661 GB I1010 20:59:18.528216 4671 mem_tracker.cc:142] MemTracker: soft memory limit is 7.066597 GB I1010 20:59:18.529121 4671 tablet_server_main.cc:54] Initializing tablet server... I1010 20:59:18.529304 4671 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 990115 I1010 20:59:18.529917 4671 server_base.cc:167] Could not load existing FS layout: Not found: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateTableWithDeadTServers.1476133114077700-1686/minicluster-data/ts-1/instance: No such file or directory (error 2) I1010 20:59:18.529927 4671 server_base.cc:168] Creating new FS layout I1010 20:59:18.530900 4671 fs_manager.cc:354] Generated new instance metadata in path /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateTableWithDeadTServers.1476133114077700-1686/minicluster-data/ts-1/instance: uuid: "b875591ca76f403cb24fdc708bf09ffe" format_stamp: "Formatted at 2016-10-10 20:59:18 on dist-test-slave-dist-test-slave-ymsa" I1010 20:59:18.531147 4549 tablet_server_main.cc:60] Tablet server successfully started. I1010 20:59:18.531503 4671 fs_manager.cc:243] Opened local filesystem: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateTableWithDeadTServers.1476133114077700-1686/minicluster-data/ts-1 uuid: "b875591ca76f403cb24fdc708bf09ffe" format_stamp: "Formatted at 2016-10-10 20:59:18 on dist-test-slave-dist-test-slave-ymsa" I1010 20:59:18.539289 4670 heartbeater.cc:292] Connected to a master server at 127.0.0.1:33283 I1010 20:59:18.539304 4670 heartbeater.cc:359] Registering TS with master... I1010 20:59:18.539314 4670 heartbeater.cc:373] Master 127.0.0.1:33283 requested a full tablet report, sending... I1010 20:59:18.539419 4512 ts_manager.cc:76] Registered new tserver with Master: d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) I1010 20:59:18.550889 4671 tablet_server_main.cc:57] Starting tablet server... I1010 20:59:18.558497 4671 rpc_server.cc:164] RPC server started. Bound to: 127.6.150.1:44212 I1010 20:59:18.558533 4671 webserver.cc:125] Starting webserver on 127.6.150.1:0 I1010 20:59:18.558537 4671 webserver.cc:136] Document root disabled I1010 20:59:18.558607 4671 webserver.cc:220] Webserver started. Bound to: http://127.6.150.1:36208/ I1010 20:59:18.558701 4671 server_base.cc:239] Dumped server information to /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateTableWithDeadTServers.1476133114077700-1686/minicluster-data/ts-1/info.pb I1010 20:59:18.564354 1686 external_mini_cluster.cc:622] Started /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver as pid 4671 I1010 20:59:18.564419 1686 external_mini_cluster.cc:586] Running /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateTableWithDeadTServers.1476133114077700-1686/minicluster-data/ts-2 --fs_data_dirs=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateTableWithDeadTServers.1476133114077700-1686/minicluster-data/ts-2 --rpc_bind_addresses=127.6.150.2:0 --local_ip_for_outbound_sockets=127.6.150.2 --webserver_interface=127.6.150.2 --webserver_port=0 --tserver_master_addrs=127.0.0.1:33283 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateTableWithDeadTServers.1476133114077700-1686/minicluster-data/ts-2 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateTableWithDeadTServers.1476133114077700-1686/minicluster-data/ts-2/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 --unlock_experimental_flags --unlock_unsafe_flags WARNING: Logging before InitGoogleLogging() is written to STDERR W1010 20:59:18.568583 4789 flags.cc:294] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1010 20:59:18.568639 4789 flags.cc:294] Enabled unsafe flag: --never_fsync=true W1010 20:59:18.568991 4789 flags.cc:294] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.150.2 I1010 20:59:18.569111 4789 mem_tracker.cc:140] MemTracker: hard memory limit is 11.777661 GB I1010 20:59:18.569121 4789 mem_tracker.cc:142] MemTracker: soft memory limit is 7.066597 GB I1010 20:59:18.570050 4789 tablet_server_main.cc:54] Initializing tablet server... I1010 20:59:18.570178 4789 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 990115 I1010 20:59:18.570845 4789 server_base.cc:167] Could not load existing FS layout: Not found: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateTableWithDeadTServers.1476133114077700-1686/minicluster-data/ts-2/instance: No such file or directory (error 2) I1010 20:59:18.570845 4671 tablet_server_main.cc:60] Tablet server successfully started. I1010 20:59:18.570855 4789 server_base.cc:168] Creating new FS layout I1010 20:59:18.571854 4788 heartbeater.cc:292] Connected to a master server at 127.0.0.1:33283 I1010 20:59:18.571885 4788 heartbeater.cc:359] Registering TS with master... I1010 20:59:18.571914 4788 heartbeater.cc:373] Master 127.0.0.1:33283 requested a full tablet report, sending... I1010 20:59:18.571990 4789 fs_manager.cc:354] Generated new instance metadata in path /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateTableWithDeadTServers.1476133114077700-1686/minicluster-data/ts-2/instance: uuid: "56563ca504254721af39bb1b55e0af86" format_stamp: "Formatted at 2016-10-10 20:59:18 on dist-test-slave-dist-test-slave-ymsa" I1010 20:59:18.572005 4512 ts_manager.cc:76] Registered new tserver with Master: b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) I1010 20:59:18.572556 4789 fs_manager.cc:243] Opened local filesystem: /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateTableWithDeadTServers.1476133114077700-1686/minicluster-data/ts-2 uuid: "56563ca504254721af39bb1b55e0af86" format_stamp: "Formatted at 2016-10-10 20:59:18 on dist-test-slave-dist-test-slave-ymsa" I1010 20:59:18.574828 4789 tablet_server_main.cc:57] Starting tablet server... I1010 20:59:18.583186 4789 rpc_server.cc:164] RPC server started. Bound to: 127.6.150.2:41850 I1010 20:59:18.583221 4789 webserver.cc:125] Starting webserver on 127.6.150.2:0 I1010 20:59:18.583225 4789 webserver.cc:136] Document root disabled I1010 20:59:18.583317 4789 webserver.cc:220] Webserver started. Bound to: http://127.6.150.2:45935/ I1010 20:59:18.583451 4789 server_base.cc:239] Dumped server information to /tmp/kudutest-1000/create-table-itest.CreateTableITest.TestCreateTableWithDeadTServers.1476133114077700-1686/minicluster-data/ts-2/info.pb I1010 20:59:18.583658 4789 tablet_server_main.cc:60] Tablet server successfully started. I1010 20:59:18.584388 4905 heartbeater.cc:292] Connected to a master server at 127.0.0.1:33283 I1010 20:59:18.584405 4905 heartbeater.cc:359] Registering TS with master... I1010 20:59:18.584419 4905 heartbeater.cc:373] Master 127.0.0.1:33283 requested a full tablet report, sending... I1010 20:59:18.584507 4512 ts_manager.cc:76] Registered new tserver with Master: 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) I1010 20:59:18.584893 1686 external_mini_cluster.cc:622] Started /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver as pid 4789 I1010 20:59:18.585628 1686 external_mini_cluster.cc:341] 3 TS(s) registered with all masters I1010 20:59:18.587211 1686 external_mini_cluster.cc:687] Killing /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver with pid 4549 I1010 20:59:18.588821 1686 external_mini_cluster.cc:687] Killing /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver with pid 4671 I1010 20:59:18.590306 1686 external_mini_cluster.cc:687] Killing /tmp/run_tha_testXiwMYW/build/release/bin/kudu-tserver with pid 4789 I1010 20:59:18.592829 4512 catalog_manager.cc:789] CreateTable from {real_user=slave, eff_user=} at 127.0.0.1:51300: name: "test" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W1010 20:59:18.594099 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:18.594113 4498 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 017f3e822a354318bfbd0c20954f50cc on TS b875591ca76f403cb24fdc708bf09ffe with a delay of 60ms (attempt = 1)... W1010 20:59:18.594529 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) W1010 20:59:18.594537 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:18.594550 4497 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 017f3e822a354318bfbd0c20954f50cc on TS 56563ca504254721af39bb1b55e0af86 with a delay of 54ms (attempt = 1)... I1010 20:59:18.594540 4496 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 017f3e822a354318bfbd0c20954f50cc on TS d28b59bf702d4784a663d8e6458ffe07 with a delay of 35ms (attempt = 1)... W1010 20:59:18.629719 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:18.629730 4496 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 017f3e822a354318bfbd0c20954f50cc on TS d28b59bf702d4784a663d8e6458ffe07 with a delay of 33ms (attempt = 2)... W1010 20:59:18.648808 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:18.648819 4497 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 017f3e822a354318bfbd0c20954f50cc on TS 56563ca504254721af39bb1b55e0af86 with a delay of 58ms (attempt = 2)... W1010 20:59:18.654353 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:18.654373 4498 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 017f3e822a354318bfbd0c20954f50cc on TS b875591ca76f403cb24fdc708bf09ffe with a delay of 65ms (attempt = 2)... W1010 20:59:18.662880 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:18.662894 4496 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 017f3e822a354318bfbd0c20954f50cc on TS d28b59bf702d4784a663d8e6458ffe07 with a delay of 83ms (attempt = 3)... W1010 20:59:18.707247 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:18.707258 4497 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 017f3e822a354318bfbd0c20954f50cc on TS 56563ca504254721af39bb1b55e0af86 with a delay of 97ms (attempt = 3)... W1010 20:59:18.719749 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:18.719760 4498 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 017f3e822a354318bfbd0c20954f50cc on TS b875591ca76f403cb24fdc708bf09ffe with a delay of 107ms (attempt = 3)... W1010 20:59:18.746054 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:18.746068 4496 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 017f3e822a354318bfbd0c20954f50cc on TS d28b59bf702d4784a663d8e6458ffe07 with a delay of 132ms (attempt = 4)... W1010 20:59:18.804664 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:18.804674 4497 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 017f3e822a354318bfbd0c20954f50cc on TS 56563ca504254721af39bb1b55e0af86 with a delay of 134ms (attempt = 4)... W1010 20:59:18.827160 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:18.827170 4498 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 017f3e822a354318bfbd0c20954f50cc on TS b875591ca76f403cb24fdc708bf09ffe with a delay of 159ms (attempt = 4)... W1010 20:59:18.878212 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:18.878238 4496 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 017f3e822a354318bfbd0c20954f50cc on TS d28b59bf702d4784a663d8e6458ffe07 with a delay of 281ms (attempt = 5)... W1010 20:59:18.939074 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:18.939085 4497 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 017f3e822a354318bfbd0c20954f50cc on TS 56563ca504254721af39bb1b55e0af86 with a delay of 272ms (attempt = 5)... W1010 20:59:18.986629 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:18.986641 4498 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 017f3e822a354318bfbd0c20954f50cc on TS b875591ca76f403cb24fdc708bf09ffe with a delay of 267ms (attempt = 5)... W1010 20:59:19.159409 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:19.159471 4496 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 017f3e822a354318bfbd0c20954f50cc on TS d28b59bf702d4784a663d8e6458ffe07 with a delay of 424ms (attempt = 6)... W1010 20:59:19.211771 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:19.211786 4497 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 017f3e822a354318bfbd0c20954f50cc on TS 56563ca504254721af39bb1b55e0af86 with a delay of 372ms (attempt = 6)... W1010 20:59:19.254428 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:19.254437 4498 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 017f3e822a354318bfbd0c20954f50cc on TS b875591ca76f403cb24fdc708bf09ffe with a delay of 329ms (attempt = 6)... W1010 20:59:19.583678 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) W1010 20:59:19.583708 4496 catalog_manager.cc:2352] Request timed out: CreateTablet RPC for tablet 017f3e822a354318bfbd0c20954f50cc on TS d28b59bf702d4784a663d8e6458ffe07 W1010 20:59:19.584249 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) W1010 20:59:19.584257 4498 catalog_manager.cc:2352] Request timed out: CreateTablet RPC for tablet 017f3e822a354318bfbd0c20954f50cc on TS b875591ca76f403cb24fdc708bf09ffe W1010 20:59:19.584632 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) W1010 20:59:19.584650 4497 catalog_manager.cc:2352] Request timed out: CreateTablet RPC for tablet 017f3e822a354318bfbd0c20954f50cc on TS 56563ca504254721af39bb1b55e0af86 W1010 20:59:19.594108 4562 catalog_manager.cc:2975] Tablet 017f3e822a354318bfbd0c20954f50cc (table test [id=2914731f866e47ad82213516818cfda0]) was not created within the allowed timeout. Replacing with a new tablet 7bad6c8ea22e4974be4ee6cab57e0d5a I1010 20:59:19.600724 4562 catalog_manager.cc:2864] Sending DeleteTablet for 3 replicas of tablet 017f3e822a354318bfbd0c20954f50cc I1010 20:59:19.600741 4562 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 017f3e822a354318bfbd0c20954f50cc on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 7bad6c8ea22e4974be4ee6cab57e0d5a at 2016-10-10 20:59:19 UTC) I1010 20:59:19.600752 4562 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 017f3e822a354318bfbd0c20954f50cc on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 7bad6c8ea22e4974be4ee6cab57e0d5a at 2016-10-10 20:59:19 UTC) I1010 20:59:19.600769 4562 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 017f3e822a354318bfbd0c20954f50cc on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 7bad6c8ea22e4974be4ee6cab57e0d5a at 2016-10-10 20:59:19 UTC) W1010 20:59:19.601032 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) W1010 20:59:19.601055 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:19.601086 4496 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on TS d28b59bf702d4784a663d8e6458ffe07 with a delay of 47ms (attempt = 1)... I1010 20:59:19.601100 4497 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on TS 56563ca504254721af39bb1b55e0af86 with a delay of 18ms (attempt = 1)... W1010 20:59:19.601106 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:19.601109 4497 catalog_manager.cc:2355] Scheduling retry of 017f3e822a354318bfbd0c20954f50cc Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 40ms (attempt = 1)... W1010 20:59:19.601107 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:19.601122 4496 catalog_manager.cc:2355] Scheduling retry of 017f3e822a354318bfbd0c20954f50cc Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 23ms (attempt = 1)... W1010 20:59:19.601065 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:19.601207 4498 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on TS b875591ca76f403cb24fdc708bf09ffe with a delay of 34ms (attempt = 1)... W1010 20:59:19.601217 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:19.601218 4498 catalog_manager.cc:2355] Scheduling retry of 017f3e822a354318bfbd0c20954f50cc Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 32ms (attempt = 1)... W1010 20:59:19.619359 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:19.619369 4497 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on TS 56563ca504254721af39bb1b55e0af86 with a delay of 65ms (attempt = 2)... I1010 20:59:19.624174 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 017f3e822a354318bfbd0c20954f50cc on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 7bad6c8ea22e4974be4ee6cab57e0d5a at 2016-10-10 20:59:19 UTC) W1010 20:59:19.624289 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:19.624299 4496 catalog_manager.cc:2355] Scheduling retry of 017f3e822a354318bfbd0c20954f50cc Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 59ms (attempt = 2)... I1010 20:59:19.633366 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 017f3e822a354318bfbd0c20954f50cc on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 7bad6c8ea22e4974be4ee6cab57e0d5a at 2016-10-10 20:59:19 UTC) W1010 20:59:19.633468 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:19.633477 4498 catalog_manager.cc:2355] Scheduling retry of 017f3e822a354318bfbd0c20954f50cc Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 69ms (attempt = 2)... W1010 20:59:19.635737 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:19.635746 4498 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on TS b875591ca76f403cb24fdc708bf09ffe with a delay of 54ms (attempt = 2)... I1010 20:59:19.641513 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 017f3e822a354318bfbd0c20954f50cc on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 7bad6c8ea22e4974be4ee6cab57e0d5a at 2016-10-10 20:59:19 UTC) W1010 20:59:19.641611 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:19.641618 4497 catalog_manager.cc:2355] Scheduling retry of 017f3e822a354318bfbd0c20954f50cc Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 55ms (attempt = 2)... W1010 20:59:19.648282 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:19.648303 4496 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on TS d28b59bf702d4784a663d8e6458ffe07 with a delay of 39ms (attempt = 2)... I1010 20:59:19.683329 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 017f3e822a354318bfbd0c20954f50cc on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 7bad6c8ea22e4974be4ee6cab57e0d5a at 2016-10-10 20:59:19 UTC) W1010 20:59:19.683456 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:19.683492 4496 catalog_manager.cc:2355] Scheduling retry of 017f3e822a354318bfbd0c20954f50cc Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 76ms (attempt = 3)... W1010 20:59:19.684880 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:19.684891 4497 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on TS 56563ca504254721af39bb1b55e0af86 with a delay of 89ms (attempt = 3)... W1010 20:59:19.687433 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:19.687444 4496 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on TS d28b59bf702d4784a663d8e6458ffe07 with a delay of 97ms (attempt = 3)... W1010 20:59:19.689990 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:19.689999 4498 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on TS b875591ca76f403cb24fdc708bf09ffe with a delay of 111ms (attempt = 3)... I1010 20:59:19.697183 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 017f3e822a354318bfbd0c20954f50cc on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 7bad6c8ea22e4974be4ee6cab57e0d5a at 2016-10-10 20:59:19 UTC) W1010 20:59:19.697336 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:19.697345 4497 catalog_manager.cc:2355] Scheduling retry of 017f3e822a354318bfbd0c20954f50cc Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 110ms (attempt = 3)... I1010 20:59:19.703272 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 017f3e822a354318bfbd0c20954f50cc on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 7bad6c8ea22e4974be4ee6cab57e0d5a at 2016-10-10 20:59:19 UTC) W1010 20:59:19.703389 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:19.703402 4498 catalog_manager.cc:2355] Scheduling retry of 017f3e822a354318bfbd0c20954f50cc Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 82ms (attempt = 3)... I1010 20:59:19.759539 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 017f3e822a354318bfbd0c20954f50cc on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 7bad6c8ea22e4974be4ee6cab57e0d5a at 2016-10-10 20:59:19 UTC) W1010 20:59:19.759639 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:19.759667 4496 catalog_manager.cc:2355] Scheduling retry of 017f3e822a354318bfbd0c20954f50cc Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 169ms (attempt = 4)... W1010 20:59:19.774626 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:19.774647 4497 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on TS 56563ca504254721af39bb1b55e0af86 with a delay of 131ms (attempt = 4)... W1010 20:59:19.784584 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:19.784597 4496 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on TS d28b59bf702d4784a663d8e6458ffe07 with a delay of 154ms (attempt = 4)... I1010 20:59:19.785593 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 017f3e822a354318bfbd0c20954f50cc on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 7bad6c8ea22e4974be4ee6cab57e0d5a at 2016-10-10 20:59:19 UTC) W1010 20:59:19.785790 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:19.785815 4498 catalog_manager.cc:2355] Scheduling retry of 017f3e822a354318bfbd0c20954f50cc Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 152ms (attempt = 4)... W1010 20:59:19.801136 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:19.801147 4498 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on TS b875591ca76f403cb24fdc708bf09ffe with a delay of 158ms (attempt = 4)... I1010 20:59:19.807940 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 017f3e822a354318bfbd0c20954f50cc on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 7bad6c8ea22e4974be4ee6cab57e0d5a at 2016-10-10 20:59:19 UTC) W1010 20:59:19.808051 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:19.808063 4497 catalog_manager.cc:2355] Scheduling retry of 017f3e822a354318bfbd0c20954f50cc Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 171ms (attempt = 4)... W1010 20:59:19.906512 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:19.906525 4497 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on TS 56563ca504254721af39bb1b55e0af86 with a delay of 291ms (attempt = 5)... I1010 20:59:19.928726 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 017f3e822a354318bfbd0c20954f50cc on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 7bad6c8ea22e4974be4ee6cab57e0d5a at 2016-10-10 20:59:19 UTC) W1010 20:59:19.928874 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:19.928921 4496 catalog_manager.cc:2355] Scheduling retry of 017f3e822a354318bfbd0c20954f50cc Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 304ms (attempt = 5)... I1010 20:59:19.938516 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 017f3e822a354318bfbd0c20954f50cc on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 7bad6c8ea22e4974be4ee6cab57e0d5a at 2016-10-10 20:59:19 UTC) W1010 20:59:19.938658 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:19.938693 4498 catalog_manager.cc:2355] Scheduling retry of 017f3e822a354318bfbd0c20954f50cc Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 261ms (attempt = 5)... W1010 20:59:19.938824 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:19.938851 4496 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on TS d28b59bf702d4784a663d8e6458ffe07 with a delay of 286ms (attempt = 5)... W1010 20:59:19.959954 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:19.959966 4498 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on TS b875591ca76f403cb24fdc708bf09ffe with a delay of 285ms (attempt = 5)... I1010 20:59:19.979713 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 017f3e822a354318bfbd0c20954f50cc on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 7bad6c8ea22e4974be4ee6cab57e0d5a at 2016-10-10 20:59:19 UTC) W1010 20:59:19.979862 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:19.979873 4497 catalog_manager.cc:2355] Scheduling retry of 017f3e822a354318bfbd0c20954f50cc Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 265ms (attempt = 5)... W1010 20:59:20.198648 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:20.198668 4497 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on TS 56563ca504254721af39bb1b55e0af86 with a delay of 392ms (attempt = 6)... I1010 20:59:20.200687 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 017f3e822a354318bfbd0c20954f50cc on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 7bad6c8ea22e4974be4ee6cab57e0d5a at 2016-10-10 20:59:19 UTC) W1010 20:59:20.200963 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:20.200974 4498 catalog_manager.cc:2355] Scheduling retry of 017f3e822a354318bfbd0c20954f50cc Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 522ms (attempt = 6)... W1010 20:59:20.225003 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:20.225044 4496 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on TS d28b59bf702d4784a663d8e6458ffe07 with a delay of 365ms (attempt = 6)... I1010 20:59:20.232996 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 017f3e822a354318bfbd0c20954f50cc on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 7bad6c8ea22e4974be4ee6cab57e0d5a at 2016-10-10 20:59:19 UTC) W1010 20:59:20.233122 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:20.233137 4496 catalog_manager.cc:2355] Scheduling retry of 017f3e822a354318bfbd0c20954f50cc Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 536ms (attempt = 6)... W1010 20:59:20.245270 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:20.245282 4498 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on TS b875591ca76f403cb24fdc708bf09ffe with a delay of 345ms (attempt = 6)... I1010 20:59:20.245898 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 017f3e822a354318bfbd0c20954f50cc on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 7bad6c8ea22e4974be4ee6cab57e0d5a at 2016-10-10 20:59:19 UTC) W1010 20:59:20.246011 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:20.246022 4497 catalog_manager.cc:2355] Scheduling retry of 017f3e822a354318bfbd0c20954f50cc Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 542ms (attempt = 6)... W1010 20:59:20.590284 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) W1010 20:59:20.590351 4496 catalog_manager.cc:2352] Request timed out: CreateTablet RPC for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on TS d28b59bf702d4784a663d8e6458ffe07 W1010 20:59:20.591095 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) W1010 20:59:20.591107 4498 catalog_manager.cc:2352] Request timed out: CreateTablet RPC for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on TS b875591ca76f403cb24fdc708bf09ffe W1010 20:59:20.591845 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) W1010 20:59:20.591856 4497 catalog_manager.cc:2352] Request timed out: CreateTablet RPC for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on TS 56563ca504254721af39bb1b55e0af86 W1010 20:59:20.600891 4562 catalog_manager.cc:2975] Tablet 7bad6c8ea22e4974be4ee6cab57e0d5a (table test [id=2914731f866e47ad82213516818cfda0]) was not created within the allowed timeout. Replacing with a new tablet 1f59d8541a3f41fb8a6c6b0de782eed3 I1010 20:59:20.601801 4562 catalog_manager.cc:2864] Sending DeleteTablet for 3 replicas of tablet 7bad6c8ea22e4974be4ee6cab57e0d5a I1010 20:59:20.601817 4562 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 1f59d8541a3f41fb8a6c6b0de782eed3 at 2016-10-10 20:59:20 UTC) I1010 20:59:20.601831 4562 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 1f59d8541a3f41fb8a6c6b0de782eed3 at 2016-10-10 20:59:20 UTC) I1010 20:59:20.601840 4562 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 1f59d8541a3f41fb8a6c6b0de782eed3 at 2016-10-10 20:59:20 UTC) W1010 20:59:20.602046 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) W1010 20:59:20.602049 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:20.602064 4498 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on TS b875591ca76f403cb24fdc708bf09ffe with a delay of 46ms (attempt = 1)... I1010 20:59:20.602074 4497 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on TS 56563ca504254721af39bb1b55e0af86 with a delay of 42ms (attempt = 1)... W1010 20:59:20.602082 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) W1010 20:59:20.602082 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:20.602090 4497 catalog_manager.cc:2355] Scheduling retry of 7bad6c8ea22e4974be4ee6cab57e0d5a Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 49ms (attempt = 1)... I1010 20:59:20.602093 4498 catalog_manager.cc:2355] Scheduling retry of 7bad6c8ea22e4974be4ee6cab57e0d5a Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 55ms (attempt = 1)... W1010 20:59:20.602118 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:20.602128 4496 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on TS d28b59bf702d4784a663d8e6458ffe07 with a delay of 56ms (attempt = 1)... W1010 20:59:20.602134 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:20.602140 4496 catalog_manager.cc:2355] Scheduling retry of 7bad6c8ea22e4974be4ee6cab57e0d5a Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 24ms (attempt = 1)... I1010 20:59:20.626214 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 1f59d8541a3f41fb8a6c6b0de782eed3 at 2016-10-10 20:59:20 UTC) W1010 20:59:20.626384 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:20.626453 4496 catalog_manager.cc:2355] Scheduling retry of 7bad6c8ea22e4974be4ee6cab57e0d5a Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 55ms (attempt = 2)... W1010 20:59:20.644327 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:20.644338 4497 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on TS 56563ca504254721af39bb1b55e0af86 with a delay of 70ms (attempt = 2)... W1010 20:59:20.648371 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:20.648383 4498 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on TS b875591ca76f403cb24fdc708bf09ffe with a delay of 39ms (attempt = 2)... I1010 20:59:20.651516 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 1f59d8541a3f41fb8a6c6b0de782eed3 at 2016-10-10 20:59:20 UTC) W1010 20:59:20.651679 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:20.651703 4497 catalog_manager.cc:2355] Scheduling retry of 7bad6c8ea22e4974be4ee6cab57e0d5a Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 75ms (attempt = 2)... I1010 20:59:20.657534 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 1f59d8541a3f41fb8a6c6b0de782eed3 at 2016-10-10 20:59:20 UTC) W1010 20:59:20.657721 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:20.657733 4498 catalog_manager.cc:2355] Scheduling retry of 7bad6c8ea22e4974be4ee6cab57e0d5a Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 63ms (attempt = 2)... W1010 20:59:20.658319 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:20.658334 4496 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on TS d28b59bf702d4784a663d8e6458ffe07 with a delay of 42ms (attempt = 2)... I1010 20:59:20.681504 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 1f59d8541a3f41fb8a6c6b0de782eed3 at 2016-10-10 20:59:20 UTC) W1010 20:59:20.681640 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:20.681670 4496 catalog_manager.cc:2355] Scheduling retry of 7bad6c8ea22e4974be4ee6cab57e0d5a Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 84ms (attempt = 3)... W1010 20:59:20.688073 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:20.688086 4498 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on TS b875591ca76f403cb24fdc708bf09ffe with a delay of 69ms (attempt = 3)... W1010 20:59:20.700492 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:20.700507 4496 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on TS d28b59bf702d4784a663d8e6458ffe07 with a delay of 104ms (attempt = 3)... W1010 20:59:20.715261 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:20.715276 4497 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on TS 56563ca504254721af39bb1b55e0af86 with a delay of 77ms (attempt = 3)... I1010 20:59:20.721316 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 1f59d8541a3f41fb8a6c6b0de782eed3 at 2016-10-10 20:59:20 UTC) W1010 20:59:20.721568 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:20.721580 4498 catalog_manager.cc:2355] Scheduling retry of 7bad6c8ea22e4974be4ee6cab57e0d5a Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 105ms (attempt = 3)... I1010 20:59:20.723769 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 017f3e822a354318bfbd0c20954f50cc on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 7bad6c8ea22e4974be4ee6cab57e0d5a at 2016-10-10 20:59:19 UTC) W1010 20:59:20.723934 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:20.723947 4498 catalog_manager.cc:2355] Scheduling retry of 017f3e822a354318bfbd0c20954f50cc Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 1062ms (attempt = 7)... I1010 20:59:20.727434 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 1f59d8541a3f41fb8a6c6b0de782eed3 at 2016-10-10 20:59:20 UTC) W1010 20:59:20.727604 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:20.727614 4497 catalog_manager.cc:2355] Scheduling retry of 7bad6c8ea22e4974be4ee6cab57e0d5a Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 84ms (attempt = 3)... W1010 20:59:20.758249 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:20.758261 4498 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on TS b875591ca76f403cb24fdc708bf09ffe with a delay of 149ms (attempt = 4)... I1010 20:59:20.765763 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 1f59d8541a3f41fb8a6c6b0de782eed3 at 2016-10-10 20:59:20 UTC) W1010 20:59:20.765879 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:20.765915 4496 catalog_manager.cc:2355] Scheduling retry of 7bad6c8ea22e4974be4ee6cab57e0d5a Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 145ms (attempt = 4)... I1010 20:59:20.769193 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 017f3e822a354318bfbd0c20954f50cc on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 7bad6c8ea22e4974be4ee6cab57e0d5a at 2016-10-10 20:59:19 UTC) W1010 20:59:20.769363 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:20.769397 4496 catalog_manager.cc:2355] Scheduling retry of 017f3e822a354318bfbd0c20954f50cc Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 1053ms (attempt = 7)... I1010 20:59:20.788787 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 017f3e822a354318bfbd0c20954f50cc on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 7bad6c8ea22e4974be4ee6cab57e0d5a at 2016-10-10 20:59:19 UTC) W1010 20:59:20.788954 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:20.788964 4497 catalog_manager.cc:2355] Scheduling retry of 017f3e822a354318bfbd0c20954f50cc Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 1045ms (attempt = 7)... W1010 20:59:20.793287 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:20.793298 4497 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on TS 56563ca504254721af39bb1b55e0af86 with a delay of 158ms (attempt = 4)... W1010 20:59:20.804654 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:20.804693 4496 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on TS d28b59bf702d4784a663d8e6458ffe07 with a delay of 171ms (attempt = 4)... I1010 20:59:20.812460 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 1f59d8541a3f41fb8a6c6b0de782eed3 at 2016-10-10 20:59:20 UTC) W1010 20:59:20.812675 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:20.812686 4497 catalog_manager.cc:2355] Scheduling retry of 7bad6c8ea22e4974be4ee6cab57e0d5a Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 173ms (attempt = 4)... I1010 20:59:20.827569 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 1f59d8541a3f41fb8a6c6b0de782eed3 at 2016-10-10 20:59:20 UTC) W1010 20:59:20.827705 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:20.827719 4498 catalog_manager.cc:2355] Scheduling retry of 7bad6c8ea22e4974be4ee6cab57e0d5a Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 177ms (attempt = 4)... W1010 20:59:20.908176 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:20.908188 4498 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on TS b875591ca76f403cb24fdc708bf09ffe with a delay of 280ms (attempt = 5)... I1010 20:59:20.911001 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 1f59d8541a3f41fb8a6c6b0de782eed3 at 2016-10-10 20:59:20 UTC) W1010 20:59:20.911178 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:20.911212 4496 catalog_manager.cc:2355] Scheduling retry of 7bad6c8ea22e4974be4ee6cab57e0d5a Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 294ms (attempt = 5)... W1010 20:59:20.952297 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:20.952337 4497 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on TS 56563ca504254721af39bb1b55e0af86 with a delay of 260ms (attempt = 5)... W1010 20:59:20.975863 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:20.975878 4496 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on TS d28b59bf702d4784a663d8e6458ffe07 with a delay of 292ms (attempt = 5)... I1010 20:59:20.986497 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 1f59d8541a3f41fb8a6c6b0de782eed3 at 2016-10-10 20:59:20 UTC) W1010 20:59:20.986801 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:20.986812 4497 catalog_manager.cc:2355] Scheduling retry of 7bad6c8ea22e4974be4ee6cab57e0d5a Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 275ms (attempt = 5)... I1010 20:59:21.005553 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 1f59d8541a3f41fb8a6c6b0de782eed3 at 2016-10-10 20:59:20 UTC) W1010 20:59:21.005789 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:21.005801 4498 catalog_manager.cc:2355] Scheduling retry of 7bad6c8ea22e4974be4ee6cab57e0d5a Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 288ms (attempt = 5)... W1010 20:59:21.189314 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:21.189333 4498 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on TS b875591ca76f403cb24fdc708bf09ffe with a delay of 402ms (attempt = 6)... I1010 20:59:21.205260 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 1f59d8541a3f41fb8a6c6b0de782eed3 at 2016-10-10 20:59:20 UTC) W1010 20:59:21.205404 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:21.205417 4496 catalog_manager.cc:2355] Scheduling retry of 7bad6c8ea22e4974be4ee6cab57e0d5a Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 522ms (attempt = 6)... W1010 20:59:21.212512 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:21.212523 4497 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on TS 56563ca504254721af39bb1b55e0af86 with a delay of 379ms (attempt = 6)... I1010 20:59:21.262691 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 1f59d8541a3f41fb8a6c6b0de782eed3 at 2016-10-10 20:59:20 UTC) W1010 20:59:21.262809 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:21.262820 4497 catalog_manager.cc:2355] Scheduling retry of 7bad6c8ea22e4974be4ee6cab57e0d5a Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 542ms (attempt = 6)... W1010 20:59:21.268061 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:21.268075 4496 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on TS d28b59bf702d4784a663d8e6458ffe07 with a delay of 323ms (attempt = 6)... I1010 20:59:21.294805 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 1f59d8541a3f41fb8a6c6b0de782eed3 at 2016-10-10 20:59:20 UTC) W1010 20:59:21.295019 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:21.295030 4498 catalog_manager.cc:2355] Scheduling retry of 7bad6c8ea22e4974be4ee6cab57e0d5a Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 525ms (attempt = 6)... W1010 20:59:21.591248 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) W1010 20:59:21.591264 4496 catalog_manager.cc:2352] Request timed out: CreateTablet RPC for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on TS d28b59bf702d4784a663d8e6458ffe07 W1010 20:59:21.591708 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) W1010 20:59:21.591720 4498 catalog_manager.cc:2352] Request timed out: CreateTablet RPC for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on TS b875591ca76f403cb24fdc708bf09ffe W1010 20:59:21.592800 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) W1010 20:59:21.592814 4497 catalog_manager.cc:2352] Request timed out: CreateTablet RPC for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on TS 56563ca504254721af39bb1b55e0af86 W1010 20:59:21.601964 4562 catalog_manager.cc:2975] Tablet 1f59d8541a3f41fb8a6c6b0de782eed3 (table test [id=2914731f866e47ad82213516818cfda0]) was not created within the allowed timeout. Replacing with a new tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 I1010 20:59:21.602953 4562 catalog_manager.cc:2864] Sending DeleteTablet for 3 replicas of tablet 1f59d8541a3f41fb8a6c6b0de782eed3 I1010 20:59:21.603008 4562 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 at 2016-10-10 20:59:21 UTC) I1010 20:59:21.603025 4562 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 at 2016-10-10 20:59:21 UTC) I1010 20:59:21.603035 4562 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 at 2016-10-10 20:59:21 UTC) W1010 20:59:21.603175 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:21.603207 4496 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on TS d28b59bf702d4784a663d8e6458ffe07 with a delay of 65ms (attempt = 1)... W1010 20:59:21.603230 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:21.603242 4496 catalog_manager.cc:2355] Scheduling retry of 1f59d8541a3f41fb8a6c6b0de782eed3 Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 65ms (attempt = 1)... W1010 20:59:21.603297 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:21.603314 4497 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on TS 56563ca504254721af39bb1b55e0af86 with a delay of 17ms (attempt = 1)... W1010 20:59:21.603324 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:21.603345 4497 catalog_manager.cc:2355] Scheduling retry of 1f59d8541a3f41fb8a6c6b0de782eed3 Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 55ms (attempt = 1)... W1010 20:59:21.603405 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:21.603415 4498 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on TS b875591ca76f403cb24fdc708bf09ffe with a delay of 30ms (attempt = 1)... W1010 20:59:21.603423 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:21.603427 4498 catalog_manager.cc:2355] Scheduling retry of 1f59d8541a3f41fb8a6c6b0de782eed3 Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 58ms (attempt = 1)... W1010 20:59:21.620589 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:21.620604 4497 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on TS 56563ca504254721af39bb1b55e0af86 with a delay of 62ms (attempt = 2)... W1010 20:59:21.633677 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:21.633694 4498 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on TS b875591ca76f403cb24fdc708bf09ffe with a delay of 68ms (attempt = 2)... I1010 20:59:21.658769 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 at 2016-10-10 20:59:21 UTC) W1010 20:59:21.659052 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:21.659063 4497 catalog_manager.cc:2355] Scheduling retry of 1f59d8541a3f41fb8a6c6b0de782eed3 Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 47ms (attempt = 2)... I1010 20:59:21.661831 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 at 2016-10-10 20:59:21 UTC) W1010 20:59:21.662014 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:21.662026 4498 catalog_manager.cc:2355] Scheduling retry of 1f59d8541a3f41fb8a6c6b0de782eed3 Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 81ms (attempt = 2)... I1010 20:59:21.668309 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 at 2016-10-10 20:59:21 UTC) W1010 20:59:21.668483 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:21.668512 4496 catalog_manager.cc:2355] Scheduling retry of 1f59d8541a3f41fb8a6c6b0de782eed3 Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 47ms (attempt = 2)... W1010 20:59:21.668527 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:21.668534 4496 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on TS d28b59bf702d4784a663d8e6458ffe07 with a delay of 68ms (attempt = 2)... W1010 20:59:21.683372 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:21.683382 4497 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on TS 56563ca504254721af39bb1b55e0af86 with a delay of 95ms (attempt = 3)... W1010 20:59:21.702406 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:21.702417 4498 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on TS b875591ca76f403cb24fdc708bf09ffe with a delay of 75ms (attempt = 3)... I1010 20:59:21.706701 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 at 2016-10-10 20:59:21 UTC) W1010 20:59:21.706895 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:21.706905 4497 catalog_manager.cc:2355] Scheduling retry of 1f59d8541a3f41fb8a6c6b0de782eed3 Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 98ms (attempt = 3)... I1010 20:59:21.715595 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 at 2016-10-10 20:59:21 UTC) W1010 20:59:21.715730 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:21.715754 4496 catalog_manager.cc:2355] Scheduling retry of 1f59d8541a3f41fb8a6c6b0de782eed3 Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 97ms (attempt = 3)... I1010 20:59:21.727483 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 1f59d8541a3f41fb8a6c6b0de782eed3 at 2016-10-10 20:59:20 UTC) W1010 20:59:21.727617 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:21.727632 4496 catalog_manager.cc:2355] Scheduling retry of 7bad6c8ea22e4974be4ee6cab57e0d5a Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 1061ms (attempt = 7)... W1010 20:59:21.736685 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:21.736701 4496 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on TS d28b59bf702d4784a663d8e6458ffe07 with a delay of 86ms (attempt = 3)... I1010 20:59:21.743577 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 at 2016-10-10 20:59:21 UTC) W1010 20:59:21.743769 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:21.743791 4498 catalog_manager.cc:2355] Scheduling retry of 1f59d8541a3f41fb8a6c6b0de782eed3 Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 101ms (attempt = 3)... W1010 20:59:21.778148 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:21.778161 4498 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on TS b875591ca76f403cb24fdc708bf09ffe with a delay of 151ms (attempt = 4)... W1010 20:59:21.779233 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:21.779245 4497 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on TS 56563ca504254721af39bb1b55e0af86 with a delay of 171ms (attempt = 4)... I1010 20:59:21.786335 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 017f3e822a354318bfbd0c20954f50cc on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 7bad6c8ea22e4974be4ee6cab57e0d5a at 2016-10-10 20:59:19 UTC) W1010 20:59:21.786610 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:21.786622 4498 catalog_manager.cc:2355] Scheduling retry of 017f3e822a354318bfbd0c20954f50cc Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 2067ms (attempt = 8)... I1010 20:59:21.805544 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 1f59d8541a3f41fb8a6c6b0de782eed3 at 2016-10-10 20:59:20 UTC) I1010 20:59:21.805567 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 at 2016-10-10 20:59:21 UTC) W1010 20:59:21.805825 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:21.805836 4497 catalog_manager.cc:2355] Scheduling retry of 1f59d8541a3f41fb8a6c6b0de782eed3 Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 170ms (attempt = 4)... W1010 20:59:21.805845 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:21.805850 4497 catalog_manager.cc:2355] Scheduling retry of 7bad6c8ea22e4974be4ee6cab57e0d5a Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 1028ms (attempt = 7)... I1010 20:59:21.812788 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 at 2016-10-10 20:59:21 UTC) W1010 20:59:21.812937 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:21.812978 4496 catalog_manager.cc:2355] Scheduling retry of 1f59d8541a3f41fb8a6c6b0de782eed3 Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 172ms (attempt = 4)... I1010 20:59:21.820960 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 1f59d8541a3f41fb8a6c6b0de782eed3 at 2016-10-10 20:59:20 UTC) W1010 20:59:21.821085 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:21.821096 4498 catalog_manager.cc:2355] Scheduling retry of 7bad6c8ea22e4974be4ee6cab57e0d5a Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 1048ms (attempt = 7)... I1010 20:59:21.822446 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 017f3e822a354318bfbd0c20954f50cc on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 7bad6c8ea22e4974be4ee6cab57e0d5a at 2016-10-10 20:59:19 UTC) W1010 20:59:21.822649 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:21.822705 4496 catalog_manager.cc:2355] Scheduling retry of 017f3e822a354318bfbd0c20954f50cc Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 2087ms (attempt = 8)... W1010 20:59:21.822918 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:21.822959 4496 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on TS d28b59bf702d4784a663d8e6458ffe07 with a delay of 137ms (attempt = 4)... I1010 20:59:21.835022 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 017f3e822a354318bfbd0c20954f50cc on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 7bad6c8ea22e4974be4ee6cab57e0d5a at 2016-10-10 20:59:19 UTC) W1010 20:59:21.835172 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:21.835186 4497 catalog_manager.cc:2355] Scheduling retry of 017f3e822a354318bfbd0c20954f50cc Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 2061ms (attempt = 8)... I1010 20:59:21.845257 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 at 2016-10-10 20:59:21 UTC) W1010 20:59:21.845378 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:21.845391 4498 catalog_manager.cc:2355] Scheduling retry of 1f59d8541a3f41fb8a6c6b0de782eed3 Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 146ms (attempt = 4)... W1010 20:59:21.929769 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:21.929781 4498 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on TS b875591ca76f403cb24fdc708bf09ffe with a delay of 259ms (attempt = 5)... W1010 20:59:21.950661 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:21.950740 4497 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on TS 56563ca504254721af39bb1b55e0af86 with a delay of 268ms (attempt = 5)... W1010 20:59:21.960142 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:21.960175 4496 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on TS d28b59bf702d4784a663d8e6458ffe07 with a delay of 275ms (attempt = 5)... I1010 20:59:21.976917 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 at 2016-10-10 20:59:21 UTC) W1010 20:59:21.977138 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:21.977150 4497 catalog_manager.cc:2355] Scheduling retry of 1f59d8541a3f41fb8a6c6b0de782eed3 Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 261ms (attempt = 5)... I1010 20:59:21.985069 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 at 2016-10-10 20:59:21 UTC) W1010 20:59:21.985239 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:21.985273 4496 catalog_manager.cc:2355] Scheduling retry of 1f59d8541a3f41fb8a6c6b0de782eed3 Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 260ms (attempt = 5)... I1010 20:59:21.991962 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 at 2016-10-10 20:59:21 UTC) W1010 20:59:21.992094 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:21.992105 4498 catalog_manager.cc:2355] Scheduling retry of 1f59d8541a3f41fb8a6c6b0de782eed3 Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 289ms (attempt = 5)... W1010 20:59:22.189746 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:22.189767 4498 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on TS b875591ca76f403cb24fdc708bf09ffe with a delay of 403ms (attempt = 6)... W1010 20:59:22.219938 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:22.219964 4497 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on TS 56563ca504254721af39bb1b55e0af86 with a delay of 373ms (attempt = 6)... W1010 20:59:22.235397 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:22.235414 4496 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on TS d28b59bf702d4784a663d8e6458ffe07 with a delay of 357ms (attempt = 6)... I1010 20:59:22.239145 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 at 2016-10-10 20:59:21 UTC) W1010 20:59:22.239349 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:22.239362 4497 catalog_manager.cc:2355] Scheduling retry of 1f59d8541a3f41fb8a6c6b0de782eed3 Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 527ms (attempt = 6)... I1010 20:59:22.245321 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 at 2016-10-10 20:59:21 UTC) W1010 20:59:22.245436 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:22.245451 4496 catalog_manager.cc:2355] Scheduling retry of 1f59d8541a3f41fb8a6c6b0de782eed3 Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 547ms (attempt = 6)... I1010 20:59:22.282130 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 at 2016-10-10 20:59:21 UTC) W1010 20:59:22.282398 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:22.282410 4498 catalog_manager.cc:2355] Scheduling retry of 1f59d8541a3f41fb8a6c6b0de782eed3 Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 549ms (attempt = 6)... W1010 20:59:22.592574 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) W1010 20:59:22.592612 4496 catalog_manager.cc:2352] Request timed out: CreateTablet RPC for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on TS d28b59bf702d4784a663d8e6458ffe07 W1010 20:59:22.593315 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) W1010 20:59:22.593328 4498 catalog_manager.cc:2352] Request timed out: CreateTablet RPC for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on TS b875591ca76f403cb24fdc708bf09ffe W1010 20:59:22.593415 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) W1010 20:59:22.593425 4497 catalog_manager.cc:2352] Request timed out: CreateTablet RPC for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on TS 56563ca504254721af39bb1b55e0af86 W1010 20:59:22.603169 4562 catalog_manager.cc:2975] Tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 (table test [id=2914731f866e47ad82213516818cfda0]) was not created within the allowed timeout. Replacing with a new tablet 53a7188e437d4c469310ea6e0f7c4faf I1010 20:59:22.603929 4562 catalog_manager.cc:2864] Sending DeleteTablet for 3 replicas of tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 I1010 20:59:22.603953 4562 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 53a7188e437d4c469310ea6e0f7c4faf at 2016-10-10 20:59:22 UTC) I1010 20:59:22.603976 4562 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 53a7188e437d4c469310ea6e0f7c4faf at 2016-10-10 20:59:22 UTC) I1010 20:59:22.603994 4562 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 53a7188e437d4c469310ea6e0f7c4faf at 2016-10-10 20:59:22 UTC) W1010 20:59:22.604064 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 53a7188e437d4c469310ea6e0f7c4faf: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:22.604092 4496 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 53a7188e437d4c469310ea6e0f7c4faf on TS d28b59bf702d4784a663d8e6458ffe07 with a delay of 53ms (attempt = 1)... W1010 20:59:22.604115 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:22.604133 4496 catalog_manager.cc:2355] Scheduling retry of 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 18ms (attempt = 1)... W1010 20:59:22.604209 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 53a7188e437d4c469310ea6e0f7c4faf: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:22.604225 4498 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 53a7188e437d4c469310ea6e0f7c4faf on TS b875591ca76f403cb24fdc708bf09ffe with a delay of 53ms (attempt = 1)... W1010 20:59:22.604233 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:22.604238 4498 catalog_manager.cc:2355] Scheduling retry of 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 28ms (attempt = 1)... W1010 20:59:22.604310 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 53a7188e437d4c469310ea6e0f7c4faf: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:22.604321 4497 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 53a7188e437d4c469310ea6e0f7c4faf on TS 56563ca504254721af39bb1b55e0af86 with a delay of 55ms (attempt = 1)... W1010 20:59:22.604328 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:22.604332 4497 catalog_manager.cc:2355] Scheduling retry of 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 26ms (attempt = 1)... I1010 20:59:22.622184 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 53a7188e437d4c469310ea6e0f7c4faf at 2016-10-10 20:59:22 UTC) W1010 20:59:22.622323 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:22.622354 4496 catalog_manager.cc:2355] Scheduling retry of 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 37ms (attempt = 2)... I1010 20:59:22.630487 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 53a7188e437d4c469310ea6e0f7c4faf at 2016-10-10 20:59:22 UTC) W1010 20:59:22.630663 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:22.630691 4497 catalog_manager.cc:2355] Scheduling retry of 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 42ms (attempt = 2)... I1010 20:59:22.632405 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 53a7188e437d4c469310ea6e0f7c4faf at 2016-10-10 20:59:22 UTC) W1010 20:59:22.632560 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:22.632570 4498 catalog_manager.cc:2355] Scheduling retry of 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 36ms (attempt = 2)... W1010 20:59:22.657317 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 53a7188e437d4c469310ea6e0f7c4faf: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:22.657371 4496 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 53a7188e437d4c469310ea6e0f7c4faf on TS d28b59bf702d4784a663d8e6458ffe07 with a delay of 43ms (attempt = 2)... W1010 20:59:22.657855 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 53a7188e437d4c469310ea6e0f7c4faf: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:22.657866 4498 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 53a7188e437d4c469310ea6e0f7c4faf on TS b875591ca76f403cb24fdc708bf09ffe with a delay of 39ms (attempt = 2)... I1010 20:59:22.659432 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 53a7188e437d4c469310ea6e0f7c4faf at 2016-10-10 20:59:22 UTC) W1010 20:59:22.659538 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:22.659565 4496 catalog_manager.cc:2355] Scheduling retry of 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 93ms (attempt = 3)... W1010 20:59:22.659975 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 53a7188e437d4c469310ea6e0f7c4faf: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:22.659988 4497 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 53a7188e437d4c469310ea6e0f7c4faf on TS 56563ca504254721af39bb1b55e0af86 with a delay of 35ms (attempt = 2)... I1010 20:59:22.669029 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 53a7188e437d4c469310ea6e0f7c4faf at 2016-10-10 20:59:22 UTC) W1010 20:59:22.669245 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:22.669256 4498 catalog_manager.cc:2355] Scheduling retry of 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 80ms (attempt = 3)... I1010 20:59:22.673140 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 53a7188e437d4c469310ea6e0f7c4faf at 2016-10-10 20:59:22 UTC) W1010 20:59:22.673321 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:22.673336 4497 catalog_manager.cc:2355] Scheduling retry of 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 106ms (attempt = 3)... W1010 20:59:22.695674 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 53a7188e437d4c469310ea6e0f7c4faf: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:22.695688 4497 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 53a7188e437d4c469310ea6e0f7c4faf on TS 56563ca504254721af39bb1b55e0af86 with a delay of 85ms (attempt = 3)... W1010 20:59:22.697677 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 53a7188e437d4c469310ea6e0f7c4faf: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:22.697692 4498 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 53a7188e437d4c469310ea6e0f7c4faf on TS b875591ca76f403cb24fdc708bf09ffe with a delay of 86ms (attempt = 3)... W1010 20:59:22.700551 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 53a7188e437d4c469310ea6e0f7c4faf: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:22.700565 4496 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 53a7188e437d4c469310ea6e0f7c4faf on TS d28b59bf702d4784a663d8e6458ffe07 with a delay of 69ms (attempt = 3)... I1010 20:59:22.749850 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 53a7188e437d4c469310ea6e0f7c4faf at 2016-10-10 20:59:22 UTC) W1010 20:59:22.749969 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:22.749979 4498 catalog_manager.cc:2355] Scheduling retry of 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 171ms (attempt = 4)... I1010 20:59:22.752619 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 53a7188e437d4c469310ea6e0f7c4faf at 2016-10-10 20:59:22 UTC) W1010 20:59:22.752806 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:22.752846 4496 catalog_manager.cc:2355] Scheduling retry of 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 157ms (attempt = 4)... I1010 20:59:22.766880 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 at 2016-10-10 20:59:21 UTC) W1010 20:59:22.767058 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:22.767069 4497 catalog_manager.cc:2355] Scheduling retry of 1f59d8541a3f41fb8a6c6b0de782eed3 Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 1035ms (attempt = 7)... W1010 20:59:22.769731 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 53a7188e437d4c469310ea6e0f7c4faf: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:22.769745 4496 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 53a7188e437d4c469310ea6e0f7c4faf on TS d28b59bf702d4784a663d8e6458ffe07 with a delay of 156ms (attempt = 4)... I1010 20:59:22.780267 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 53a7188e437d4c469310ea6e0f7c4faf at 2016-10-10 20:59:22 UTC) W1010 20:59:22.780437 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:22.780462 4497 catalog_manager.cc:2355] Scheduling retry of 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 156ms (attempt = 4)... W1010 20:59:22.781764 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 53a7188e437d4c469310ea6e0f7c4faf: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:22.781774 4497 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 53a7188e437d4c469310ea6e0f7c4faf on TS 56563ca504254721af39bb1b55e0af86 with a delay of 175ms (attempt = 4)... W1010 20:59:22.784235 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 53a7188e437d4c469310ea6e0f7c4faf: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:22.784252 4498 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 53a7188e437d4c469310ea6e0f7c4faf on TS b875591ca76f403cb24fdc708bf09ffe with a delay of 128ms (attempt = 4)... I1010 20:59:22.788704 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 1f59d8541a3f41fb8a6c6b0de782eed3 at 2016-10-10 20:59:20 UTC) W1010 20:59:22.788836 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:22.788851 4496 catalog_manager.cc:2355] Scheduling retry of 7bad6c8ea22e4974be4ee6cab57e0d5a Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 2093ms (attempt = 8)... I1010 20:59:22.792516 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 at 2016-10-10 20:59:21 UTC) W1010 20:59:22.792645 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:22.792659 4496 catalog_manager.cc:2355] Scheduling retry of 1f59d8541a3f41fb8a6c6b0de782eed3 Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 1056ms (attempt = 7)... I1010 20:59:22.831480 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 at 2016-10-10 20:59:21 UTC) W1010 20:59:22.831629 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:22.831641 4498 catalog_manager.cc:2355] Scheduling retry of 1f59d8541a3f41fb8a6c6b0de782eed3 Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 1061ms (attempt = 7)... I1010 20:59:22.834060 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 1f59d8541a3f41fb8a6c6b0de782eed3 at 2016-10-10 20:59:20 UTC) W1010 20:59:22.834190 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:22.834203 4497 catalog_manager.cc:2355] Scheduling retry of 7bad6c8ea22e4974be4ee6cab57e0d5a Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 2097ms (attempt = 8)... I1010 20:59:22.869823 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 1f59d8541a3f41fb8a6c6b0de782eed3 at 2016-10-10 20:59:20 UTC) W1010 20:59:22.870085 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:22.870097 4498 catalog_manager.cc:2355] Scheduling retry of 7bad6c8ea22e4974be4ee6cab57e0d5a Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 2049ms (attempt = 8)... I1010 20:59:22.909899 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 53a7188e437d4c469310ea6e0f7c4faf at 2016-10-10 20:59:22 UTC) W1010 20:59:22.910073 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:22.910086 4496 catalog_manager.cc:2355] Scheduling retry of 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 295ms (attempt = 5)... W1010 20:59:22.913451 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 53a7188e437d4c469310ea6e0f7c4faf: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:22.913465 4498 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 53a7188e437d4c469310ea6e0f7c4faf on TS b875591ca76f403cb24fdc708bf09ffe with a delay of 292ms (attempt = 5)... I1010 20:59:22.921632 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 53a7188e437d4c469310ea6e0f7c4faf at 2016-10-10 20:59:22 UTC) W1010 20:59:22.921782 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:22.921793 4498 catalog_manager.cc:2355] Scheduling retry of 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 259ms (attempt = 5)... W1010 20:59:22.925927 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 53a7188e437d4c469310ea6e0f7c4faf: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:22.925943 4496 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 53a7188e437d4c469310ea6e0f7c4faf on TS d28b59bf702d4784a663d8e6458ffe07 with a delay of 282ms (attempt = 5)... I1010 20:59:22.937494 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 53a7188e437d4c469310ea6e0f7c4faf at 2016-10-10 20:59:22 UTC) W1010 20:59:22.937659 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:22.937669 4497 catalog_manager.cc:2355] Scheduling retry of 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 304ms (attempt = 5)... W1010 20:59:22.957919 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 53a7188e437d4c469310ea6e0f7c4faf: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:22.957944 4497 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 53a7188e437d4c469310ea6e0f7c4faf on TS 56563ca504254721af39bb1b55e0af86 with a delay of 300ms (attempt = 5)... I1010 20:59:23.181360 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 53a7188e437d4c469310ea6e0f7c4faf at 2016-10-10 20:59:22 UTC) W1010 20:59:23.181530 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:23.181543 4498 catalog_manager.cc:2355] Scheduling retry of 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 548ms (attempt = 6)... I1010 20:59:23.205147 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 53a7188e437d4c469310ea6e0f7c4faf at 2016-10-10 20:59:22 UTC) W1010 20:59:23.205272 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:23.205297 4496 catalog_manager.cc:2355] Scheduling retry of 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 518ms (attempt = 6)... W1010 20:59:23.205921 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 53a7188e437d4c469310ea6e0f7c4faf: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:23.205937 4498 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 53a7188e437d4c469310ea6e0f7c4faf on TS b875591ca76f403cb24fdc708bf09ffe with a delay of 388ms (attempt = 6)... W1010 20:59:23.208150 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 53a7188e437d4c469310ea6e0f7c4faf: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:23.208165 4496 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 53a7188e437d4c469310ea6e0f7c4faf on TS d28b59bf702d4784a663d8e6458ffe07 with a delay of 385ms (attempt = 6)... I1010 20:59:23.242473 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 53a7188e437d4c469310ea6e0f7c4faf at 2016-10-10 20:59:22 UTC) W1010 20:59:23.242583 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:23.242596 4497 catalog_manager.cc:2355] Scheduling retry of 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 529ms (attempt = 6)... W1010 20:59:23.258894 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 53a7188e437d4c469310ea6e0f7c4faf: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:23.258908 4497 catalog_manager.cc:2355] Scheduling retry of CreateTablet RPC for tablet 53a7188e437d4c469310ea6e0f7c4faf on TS 56563ca504254721af39bb1b55e0af86 with a delay of 335ms (attempt = 6)... W1010 20:59:23.593372 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Create Tablet RPC failed for tablet 53a7188e437d4c469310ea6e0f7c4faf: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) W1010 20:59:23.593390 4496 catalog_manager.cc:2352] Request timed out: CreateTablet RPC for tablet 53a7188e437d4c469310ea6e0f7c4faf on TS d28b59bf702d4784a663d8e6458ffe07 W1010 20:59:23.594868 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Create Tablet RPC failed for tablet 53a7188e437d4c469310ea6e0f7c4faf: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) W1010 20:59:23.594882 4497 catalog_manager.cc:2352] Request timed out: CreateTablet RPC for tablet 53a7188e437d4c469310ea6e0f7c4faf on TS 56563ca504254721af39bb1b55e0af86 W1010 20:59:23.594900 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Create Tablet RPC failed for tablet 53a7188e437d4c469310ea6e0f7c4faf: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) W1010 20:59:23.594908 4498 catalog_manager.cc:2352] Request timed out: CreateTablet RPC for tablet 53a7188e437d4c469310ea6e0f7c4faf on TS b875591ca76f403cb24fdc708bf09ffe W1010 20:59:23.604106 4562 catalog_manager.cc:2975] Tablet 53a7188e437d4c469310ea6e0f7c4faf (table test [id=2914731f866e47ad82213516818cfda0]) was not created within the allowed timeout. Replacing with a new tablet 876663b889b74a1f8c581b3db671d0c1 W1010 20:59:23.604132 4562 catalog_manager.cc:3116] Aborting the current task due to error: Invalid argument: An error occured while selecting replicas for tablet 876663b889b74a1f8c581b3db671d0c1: Not enough tablet servers are online for table 'test'. Need at least 3 replicas, but only 0 tablet servers are available E1010 20:59:23.604142 4562 catalog_manager.cc:383] Error processing pending assignments, aborting the current task: Invalid argument: An error occured while selecting replicas for tablet 876663b889b74a1f8c581b3db671d0c1: Not enough tablet servers are online for table 'test'. Need at least 3 replicas, but only 0 tablet servers are available I1010 20:59:23.723350 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 53a7188e437d4c469310ea6e0f7c4faf at 2016-10-10 20:59:22 UTC) W1010 20:59:23.723511 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:23.723526 4496 catalog_manager.cc:2355] Scheduling retry of 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 1040ms (attempt = 7)... I1010 20:59:23.730263 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 53a7188e437d4c469310ea6e0f7c4faf at 2016-10-10 20:59:22 UTC) W1010 20:59:23.730378 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:23.730391 4498 catalog_manager.cc:2355] Scheduling retry of 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 1068ms (attempt = 7)... I1010 20:59:23.772260 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 53a7188e437d4c469310ea6e0f7c4faf at 2016-10-10 20:59:22 UTC) W1010 20:59:23.772404 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:23.772419 4497 catalog_manager.cc:2355] Scheduling retry of 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 1028ms (attempt = 7)... I1010 20:59:23.802706 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 at 2016-10-10 20:59:21 UTC) W1010 20:59:23.802804 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:23.802816 4497 catalog_manager.cc:2355] Scheduling retry of 1f59d8541a3f41fb8a6c6b0de782eed3 Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 2071ms (attempt = 8)... I1010 20:59:23.848700 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 at 2016-10-10 20:59:21 UTC) W1010 20:59:23.848853 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:23.848866 4496 catalog_manager.cc:2355] Scheduling retry of 1f59d8541a3f41fb8a6c6b0de782eed3 Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 2085ms (attempt = 8)... I1010 20:59:23.854701 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 017f3e822a354318bfbd0c20954f50cc on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 7bad6c8ea22e4974be4ee6cab57e0d5a at 2016-10-10 20:59:19 UTC) W1010 20:59:23.855060 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:23.855077 4498 catalog_manager.cc:2355] Scheduling retry of 017f3e822a354318bfbd0c20954f50cc Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 4129ms (attempt = 9)... I1010 20:59:23.893234 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 at 2016-10-10 20:59:21 UTC) W1010 20:59:23.893373 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:23.893388 4498 catalog_manager.cc:2355] Scheduling retry of 1f59d8541a3f41fb8a6c6b0de782eed3 Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 2082ms (attempt = 8)... I1010 20:59:23.897164 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 017f3e822a354318bfbd0c20954f50cc on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 7bad6c8ea22e4974be4ee6cab57e0d5a at 2016-10-10 20:59:19 UTC) W1010 20:59:23.897281 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:23.897295 4497 catalog_manager.cc:2355] Scheduling retry of 017f3e822a354318bfbd0c20954f50cc Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 4113ms (attempt = 9)... I1010 20:59:23.909757 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 017f3e822a354318bfbd0c20954f50cc on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 7bad6c8ea22e4974be4ee6cab57e0d5a at 2016-10-10 20:59:19 UTC) W1010 20:59:23.909910 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:23.909937 4496 catalog_manager.cc:2355] Scheduling retry of 017f3e822a354318bfbd0c20954f50cc Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 4108ms (attempt = 9)... W1010 20:59:24.604239 4562 catalog_manager.cc:2975] Tablet 53a7188e437d4c469310ea6e0f7c4faf (table test [id=2914731f866e47ad82213516818cfda0]) was not created within the allowed timeout. Replacing with a new tablet ec84aeb3dfef462d831b59cff8d10457 W1010 20:59:24.604275 4562 catalog_manager.cc:3116] Aborting the current task due to error: Invalid argument: An error occured while selecting replicas for tablet ec84aeb3dfef462d831b59cff8d10457: Not enough tablet servers are online for table 'test'. Need at least 3 replicas, but only 0 tablet servers are available E1010 20:59:24.604286 4562 catalog_manager.cc:383] Error processing pending assignments, aborting the current task: Invalid argument: An error occured while selecting replicas for tablet ec84aeb3dfef462d831b59cff8d10457: Not enough tablet servers are online for table 'test'. Need at least 3 replicas, but only 0 tablet servers are available I1010 20:59:24.763581 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 53a7188e437d4c469310ea6e0f7c4faf at 2016-10-10 20:59:22 UTC) W1010 20:59:24.763903 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:24.763936 4496 catalog_manager.cc:2355] Scheduling retry of 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 2079ms (attempt = 8)... I1010 20:59:24.799047 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 53a7188e437d4c469310ea6e0f7c4faf at 2016-10-10 20:59:22 UTC) W1010 20:59:24.799456 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:24.799468 4498 catalog_manager.cc:2355] Scheduling retry of 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 2065ms (attempt = 8)... I1010 20:59:24.801081 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 53a7188e437d4c469310ea6e0f7c4faf at 2016-10-10 20:59:22 UTC) W1010 20:59:24.801239 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:24.801250 4497 catalog_manager.cc:2355] Scheduling retry of 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 2057ms (attempt = 8)... I1010 20:59:24.881896 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 1f59d8541a3f41fb8a6c6b0de782eed3 at 2016-10-10 20:59:20 UTC) W1010 20:59:24.882076 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:24.882110 4496 catalog_manager.cc:2355] Scheduling retry of 7bad6c8ea22e4974be4ee6cab57e0d5a Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 4111ms (attempt = 9)... I1010 20:59:24.919782 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 1f59d8541a3f41fb8a6c6b0de782eed3 at 2016-10-10 20:59:20 UTC) W1010 20:59:24.919960 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:24.919972 4498 catalog_manager.cc:2355] Scheduling retry of 7bad6c8ea22e4974be4ee6cab57e0d5a Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 4102ms (attempt = 9)... I1010 20:59:24.931622 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 1f59d8541a3f41fb8a6c6b0de782eed3 at 2016-10-10 20:59:20 UTC) W1010 20:59:24.931802 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:24.931812 4497 catalog_manager.cc:2355] Scheduling retry of 7bad6c8ea22e4974be4ee6cab57e0d5a Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 4104ms (attempt = 9)... W1010 20:59:25.604399 4562 catalog_manager.cc:2975] Tablet 53a7188e437d4c469310ea6e0f7c4faf (table test [id=2914731f866e47ad82213516818cfda0]) was not created within the allowed timeout. Replacing with a new tablet 038b3c45800e4f87a173efd400d3d1ae W1010 20:59:25.604435 4562 catalog_manager.cc:3116] Aborting the current task due to error: Invalid argument: An error occured while selecting replicas for tablet 038b3c45800e4f87a173efd400d3d1ae: Not enough tablet servers are online for table 'test'. Need at least 3 replicas, but only 0 tablet servers are available E1010 20:59:25.604459 4562 catalog_manager.cc:383] Error processing pending assignments, aborting the current task: Invalid argument: An error occured while selecting replicas for tablet 038b3c45800e4f87a173efd400d3d1ae: Not enough tablet servers are online for table 'test'. Need at least 3 replicas, but only 0 tablet servers are available I1010 20:59:25.874505 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 at 2016-10-10 20:59:21 UTC) W1010 20:59:25.874737 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:25.874749 4497 catalog_manager.cc:2355] Scheduling retry of 1f59d8541a3f41fb8a6c6b0de782eed3 Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 4115ms (attempt = 9)... I1010 20:59:25.933917 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 at 2016-10-10 20:59:21 UTC) W1010 20:59:25.934150 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:25.934183 4496 catalog_manager.cc:2355] Scheduling retry of 1f59d8541a3f41fb8a6c6b0de782eed3 Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 4141ms (attempt = 9)... I1010 20:59:25.975900 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 at 2016-10-10 20:59:21 UTC) W1010 20:59:25.976037 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:25.976048 4498 catalog_manager.cc:2355] Scheduling retry of 1f59d8541a3f41fb8a6c6b0de782eed3 Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 4141ms (attempt = 9)... W1010 20:59:26.604557 4562 catalog_manager.cc:2975] Tablet 53a7188e437d4c469310ea6e0f7c4faf (table test [id=2914731f866e47ad82213516818cfda0]) was not created within the allowed timeout. Replacing with a new tablet 67f21f0b24184f9697b3cbe2213d0b25 W1010 20:59:26.604617 4562 catalog_manager.cc:3116] Aborting the current task due to error: Invalid argument: An error occured while selecting replicas for tablet 67f21f0b24184f9697b3cbe2213d0b25: Not enough tablet servers are online for table 'test'. Need at least 3 replicas, but only 0 tablet servers are available E1010 20:59:26.604629 4562 catalog_manager.cc:383] Error processing pending assignments, aborting the current task: Invalid argument: An error occured while selecting replicas for tablet 67f21f0b24184f9697b3cbe2213d0b25: Not enough tablet servers are online for table 'test'. Need at least 3 replicas, but only 0 tablet servers are available I1010 20:59:26.843008 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 53a7188e437d4c469310ea6e0f7c4faf at 2016-10-10 20:59:22 UTC) W1010 20:59:26.843291 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:26.843334 4496 catalog_manager.cc:2355] Scheduling retry of 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 4118ms (attempt = 9)... I1010 20:59:26.858551 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 53a7188e437d4c469310ea6e0f7c4faf at 2016-10-10 20:59:22 UTC) W1010 20:59:26.858799 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:26.858811 4497 catalog_manager.cc:2355] Scheduling retry of 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 4117ms (attempt = 9)... I1010 20:59:26.864708 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 53a7188e437d4c469310ea6e0f7c4faf at 2016-10-10 20:59:22 UTC) W1010 20:59:26.864886 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:26.864897 4498 catalog_manager.cc:2355] Scheduling retry of 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 4141ms (attempt = 9)... W1010 20:59:27.604722 4562 catalog_manager.cc:2975] Tablet 53a7188e437d4c469310ea6e0f7c4faf (table test [id=2914731f866e47ad82213516818cfda0]) was not created within the allowed timeout. Replacing with a new tablet 9a28b2fcbf724b9d8304ee882c5885ea W1010 20:59:27.604759 4562 catalog_manager.cc:3116] Aborting the current task due to error: Invalid argument: An error occured while selecting replicas for tablet 9a28b2fcbf724b9d8304ee882c5885ea: Not enough tablet servers are online for table 'test'. Need at least 3 replicas, but only 0 tablet servers are available E1010 20:59:27.604769 4562 catalog_manager.cc:383] Error processing pending assignments, aborting the current task: Invalid argument: An error occured while selecting replicas for tablet 9a28b2fcbf724b9d8304ee882c5885ea: Not enough tablet servers are online for table 'test'. Need at least 3 replicas, but only 0 tablet servers are available I1010 20:59:27.984931 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 017f3e822a354318bfbd0c20954f50cc on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 7bad6c8ea22e4974be4ee6cab57e0d5a at 2016-10-10 20:59:19 UTC) W1010 20:59:27.987802 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:27.987814 4498 catalog_manager.cc:2355] Scheduling retry of 017f3e822a354318bfbd0c20954f50cc Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 8211ms (attempt = 10)... I1010 20:59:28.011065 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 017f3e822a354318bfbd0c20954f50cc on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 7bad6c8ea22e4974be4ee6cab57e0d5a at 2016-10-10 20:59:19 UTC) W1010 20:59:28.011314 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:28.011325 4497 catalog_manager.cc:2355] Scheduling retry of 017f3e822a354318bfbd0c20954f50cc Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 8201ms (attempt = 10)... I1010 20:59:28.017998 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 017f3e822a354318bfbd0c20954f50cc on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 7bad6c8ea22e4974be4ee6cab57e0d5a at 2016-10-10 20:59:19 UTC) W1010 20:59:28.018122 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 017f3e822a354318bfbd0c20954f50cc: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:28.018138 4496 catalog_manager.cc:2355] Scheduling retry of 017f3e822a354318bfbd0c20954f50cc Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 8193ms (attempt = 10)... W1010 20:59:28.604864 4562 catalog_manager.cc:2975] Tablet 53a7188e437d4c469310ea6e0f7c4faf (table test [id=2914731f866e47ad82213516818cfda0]) was not created within the allowed timeout. Replacing with a new tablet 43d1f7d704ab4bd5a9d584fa60b84f30 W1010 20:59:28.604899 4562 catalog_manager.cc:3116] Aborting the current task due to error: Invalid argument: An error occured while selecting replicas for tablet 43d1f7d704ab4bd5a9d584fa60b84f30: Not enough tablet servers are online for table 'test'. Need at least 3 replicas, but only 0 tablet servers are available E1010 20:59:28.604909 4562 catalog_manager.cc:383] Error processing pending assignments, aborting the current task: Invalid argument: An error occured while selecting replicas for tablet 43d1f7d704ab4bd5a9d584fa60b84f30: Not enough tablet servers are online for table 'test'. Need at least 3 replicas, but only 0 tablet servers are available I1010 20:59:28.993175 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 1f59d8541a3f41fb8a6c6b0de782eed3 at 2016-10-10 20:59:20 UTC) W1010 20:59:28.993525 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:28.993559 4496 catalog_manager.cc:2355] Scheduling retry of 7bad6c8ea22e4974be4ee6cab57e0d5a Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 8218ms (attempt = 10)... I1010 20:59:29.022708 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 1f59d8541a3f41fb8a6c6b0de782eed3 at 2016-10-10 20:59:20 UTC) W1010 20:59:29.022950 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:29.022963 4498 catalog_manager.cc:2355] Scheduling retry of 7bad6c8ea22e4974be4ee6cab57e0d5a Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 8194ms (attempt = 10)... I1010 20:59:29.036303 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 1f59d8541a3f41fb8a6c6b0de782eed3 at 2016-10-10 20:59:20 UTC) W1010 20:59:29.036540 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 7bad6c8ea22e4974be4ee6cab57e0d5a: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:29.036551 4497 catalog_manager.cc:2355] Scheduling retry of 7bad6c8ea22e4974be4ee6cab57e0d5a Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 8213ms (attempt = 10)... W1010 20:59:29.605012 4562 catalog_manager.cc:2975] Tablet 53a7188e437d4c469310ea6e0f7c4faf (table test [id=2914731f866e47ad82213516818cfda0]) was not created within the allowed timeout. Replacing with a new tablet b9591a89e4314aebb5b4f52482fa1a1b W1010 20:59:29.605052 4562 catalog_manager.cc:3116] Aborting the current task due to error: Invalid argument: An error occured while selecting replicas for tablet b9591a89e4314aebb5b4f52482fa1a1b: Not enough tablet servers are online for table 'test'. Need at least 3 replicas, but only 0 tablet servers are available E1010 20:59:29.605062 4562 catalog_manager.cc:383] Error processing pending assignments, aborting the current task: Invalid argument: An error occured while selecting replicas for tablet b9591a89e4314aebb5b4f52482fa1a1b: Not enough tablet servers are online for table 'test'. Need at least 3 replicas, but only 0 tablet servers are available I1010 20:59:29.990252 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 at 2016-10-10 20:59:21 UTC) W1010 20:59:29.992868 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:29.992882 4497 catalog_manager.cc:2355] Scheduling retry of 1f59d8541a3f41fb8a6c6b0de782eed3 Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 8232ms (attempt = 10)... I1010 20:59:30.075259 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 at 2016-10-10 20:59:21 UTC) W1010 20:59:30.075415 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:30.075429 4496 catalog_manager.cc:2355] Scheduling retry of 1f59d8541a3f41fb8a6c6b0de782eed3 Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 8217ms (attempt = 10)... I1010 20:59:30.118060 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1f59d8541a3f41fb8a6c6b0de782eed3 on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 at 2016-10-10 20:59:21 UTC) W1010 20:59:30.118273 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 1f59d8541a3f41fb8a6c6b0de782eed3: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:30.118284 4498 catalog_manager.cc:2355] Scheduling retry of 1f59d8541a3f41fb8a6c6b0de782eed3 Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 8235ms (attempt = 10)... W1010 20:59:30.605160 4562 catalog_manager.cc:2975] Tablet 53a7188e437d4c469310ea6e0f7c4faf (table test [id=2914731f866e47ad82213516818cfda0]) was not created within the allowed timeout. Replacing with a new tablet 951281ad75ce487b834aa4b625c8cb3f W1010 20:59:30.605191 4562 catalog_manager.cc:3116] Aborting the current task due to error: Invalid argument: An error occured while selecting replicas for tablet 951281ad75ce487b834aa4b625c8cb3f: Not enough tablet servers are online for table 'test'. Need at least 3 replicas, but only 0 tablet servers are available E1010 20:59:30.605201 4562 catalog_manager.cc:383] Error processing pending assignments, aborting the current task: Invalid argument: An error occured while selecting replicas for tablet 951281ad75ce487b834aa4b625c8cb3f: Not enough tablet servers are online for table 'test'. Need at least 3 replicas, but only 0 tablet servers are available I1010 20:59:30.961391 4496 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341) (Replaced by 53a7188e437d4c469310ea6e0f7c4faf at 2016-10-10 20:59:22 UTC) W1010 20:59:30.961730 4496 catalog_manager.cc:2301] TS d28b59bf702d4784a663d8e6458ffe07 (127.6.150.0:48341): Delete Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.0:48341: connect: Connection refused (error 111) I1010 20:59:30.961792 4496 catalog_manager.cc:2355] Scheduling retry of 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 Delete Tablet RPC for TS=d28b59bf702d4784a663d8e6458ffe07 with a delay of 8214ms (attempt = 10)... I1010 20:59:30.976577 4497 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850) (Replaced by 53a7188e437d4c469310ea6e0f7c4faf at 2016-10-10 20:59:22 UTC) W1010 20:59:30.976680 4497 catalog_manager.cc:2301] TS 56563ca504254721af39bb1b55e0af86 (127.6.150.2:41850): Delete Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.2:41850: connect: Connection refused (error 111) I1010 20:59:30.976696 4497 catalog_manager.cc:2355] Scheduling retry of 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 Delete Tablet RPC for TS=56563ca504254721af39bb1b55e0af86 with a delay of 8233ms (attempt = 10)... I1010 20:59:31.006105 4498 catalog_manager.cc:2591] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 on b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212) (Replaced by 53a7188e437d4c469310ea6e0f7c4faf at 2016-10-10 20:59:22 UTC) W1010 20:59:31.006386 4498 catalog_manager.cc:2301] TS b875591ca76f403cb24fdc708bf09ffe (127.6.150.1:44212): Delete Tablet RPC failed for tablet 8ddc7df245ac4c29bfeaa7eb2f2fa6b6: Network error: Client connection negotiation failed: client connection to 127.6.150.1:44212: connect: Connection refused (error 111) I1010 20:59:31.006397 4498 catalog_manager.cc:2355] Scheduling retry of 8ddc7df245ac4c29bfeaa7eb2f2fa6b6 Delete Tablet RPC for TS=b875591ca76f403cb24fdc708bf09ffe with a delay of 8231ms (attempt = 10)... W1010 20:59:31.605309 4562 catalog_manager.cc:2975] Tablet 53a7188e437d4c469310ea6e0f7c4faf (table test [id=2914731f866e47ad82213516818cfda0]) was not created within the allowed timeout. Replacing with a new tablet 2b0bbd5c7e26432aa5cef37d0c8c6117 W1010 20:59:31.605343 4562 catalog_manager.cc:3116] Aborting the current task due to error: Invalid argument: An error occured while selecting replicas for tablet 2b0bbd5c7e26432aa5cef37d0c8c6117: Not enough tablet servers are online for table 'test'. Need at least 3 replicas, but only 0 tablet servers are available E1010 20:59:31.605351 4562 catalog_manager.cc:383] Error processing pending assignments, aborting the current task: Invalid argument: An error occured while selecting replicas for tablet 2b0bbd5c7e26432aa5cef37d0c8c6117: Not enough tablet servers are online for table 'test'. Need at least 3 replicas, but only 0 tablet servers are available W1010 20:59:32.605443 4562 catalog_manager.cc:2975] Tablet 53a7188e437d4c469310ea6e0f7c4faf (table test [id=2914731f866e47ad82213516818cfda0]) was not created within the allowed timeout. Replacing with a new tablet bbad26d906b743e3bcca99a3b857faeb W1010 20:59:32.605473 4562 catalog_manager.cc:3116] Aborting the current task due to error: Invalid argument: An error occured while selecting replicas for tablet bbad26d906b743e3bcca99a3b857faeb: Not enough tablet servers are online for table 'test'. Need at least 3 replicas, but only 0 tablet servers are available E1010 20:59:32.605482 4562 catalog_manager.cc:383] Error processing pending assignments, aborting the current task: Invalid argument: An error occured while selecting replicas for tablet bbad26d906b743e3bcca99a3b857faeb: Not enough tablet servers are online for table 'test'. Need at least 3 replicas, but only 0 tablet servers are available W1010 20:59:33.605602 4562 catalog_manager.cc:2975] Tablet 53a7188e437d4c469310ea6e0f7c4faf (table test [id=2914731f866e47ad82213516818cfda0]) was not created within the allowed timeout. Replacing with a new tablet 52965789e2d349cca2ec09058763e7e6 W1010 20:59:33.605638 4562 catalog_manager.cc:3116] Aborting the current task due to error: Invalid argument: An error occured while selecting replicas for tablet 52965789e2d349cca2ec09058763e7e6: Not enough tablet servers are online for table 'test'. Need at least 3 replicas, but only 0 tablet servers are available E1010 20:59:33.605649 4562 catalog_manager.cc:383] Error processing pending assignments, aborting the current task: Invalid argument: An error occured while selecting replicas for tablet 52965789e2d349cca2ec09058763e7e6: Not enough tablet servers are online for table 'test'. Need at least 3 replicas, but only 0 tablet servers are available I1010 20:59:33.607827 1686 external_mini_cluster.cc:687] Killing /tmp/run_tha_testXiwMYW/build/release/bin/kudu-master with pid 4491 [ OK ] CreateTableITest.TestCreateTableWithDeadTServers (15140 ms) [----------] 3 tests from CreateTableITest (59530 ms total) [----------] Global test environment tear-down [==========] 3 tests from 1 test case ran. (59530 ms total) [ PASSED ] 2 tests. [ FAILED ] 1 test, listed below: [ FAILED ] CreateTableITest.TestSpreadReplicasEvenly 1 FAILED TEST