Note: This is test shard 5 of 8. [==========] Running 5 tests from 1 test case. [----------] Global test environment set-up. [----------] 5 tests from RaftConsensusITest [ RUN ] RaftConsensusITest.TestInsertOnNonLeader I0819 06:33:32.914191 1150 test_util.cc:185] Using random seed: 1823431099 I0819 06:33:33.095290 1150 ts_itest-base.h:81] Starting cluster with: I0819 06:33:33.095453 1150 ts_itest-base.h:82] -------------- I0819 06:33:33.095531 1150 ts_itest-base.h:83] 3 tablet servers I0819 06:33:33.095614 1150 ts_itest-base.h:84] 3 replicas per TS I0819 06:33:33.095693 1150 ts_itest-base.h:85] -------------- I0819 06:33:33.099545 1150 external_mini_cluster.cc:752] Running /tmp/run_tha_test479zHR/build/tsan/bin/kudu-master /tmp/run_tha_test479zHR/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.126.254:0 --webserver_interface=127.4.126.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=flag --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0819 06:33:33.237077 1162 flags.cc:384] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0819 06:33:33.237465 1162 flags.cc:384] Enabled unsafe flag: --never_fsync=true W0819 06:33:33.253886 1162 flags.cc:384] Enabled experimental flag: --ipki_ca_key_size=1024 W0819 06:33:33.254034 1162 flags.cc:384] Enabled experimental flag: --ipki_server_key_size=1024 W0819 06:33:33.254132 1162 flags.cc:384] Enabled experimental flag: --tsk_num_rsa_bits=512 I0819 06:33:33.276808 1162 master_main.cc:60] Master server non-default flags: --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.126.254:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.4.126.254 --webserver_port=0 --never_fsync=true --heap_profile_path=/tmp/kudu-master.1162 --redact=flag --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.5.0-SNAPSHOT revision ba0f21e2316b78f167bbaba44cab392558d97b6f build type FASTDEBUG built by None at 19 Aug 2017 06:31:02 UTC on jenkins-slave-c3oe build id 9422 TSAN enabled I0819 06:33:33.287770 1162 master_main.cc:67] Initializing master server... I0819 06:33:33.289311 1162 system_ntp.cc:96] NTP initialized. Resolution in nanos?: 1 Skew PPM: 500 Current error: 1280905 I0819 06:33:33.290243 1162 env_posix.cc:1586] Not raising process file limit of 65536; it is already as high as it can go I0819 06:33:33.290518 1162 file_cache.cc:464] Constructed file cache lbm with capacity 26214 I0819 06:33:33.295495 1162 server_base.cc:215] Could not load existing FS layout: Not found: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0819 06:33:33.295641 1162 server_base.cc:216] Creating new FS layout I0819 06:33:33.303172 1162 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data/instance: uuid: "ff69923877524a91b05a9c6806bbac2f" format_stamp: "Formatted at 2017-08-19 06:33:33 on dist-test-slave-dist-test-slave-j33z" I0819 06:33:33.303908 1162 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "ff69923877524a91b05a9c6806bbac2f" format_stamp: "Formatted at 2017-08-19 06:33:33 on dist-test-slave-dist-test-slave-j33z" I0819 06:33:33.316100 1162 fs_manager.cc:280] Time spent opening block manager: real 0.005s user 0.000s sys 0.008s I0819 06:33:33.316249 1162 fs_manager.cc:283] Opened local filesystem: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data,/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/master-0/wal uuid: "ff69923877524a91b05a9c6806bbac2f" format_stamp: "Formatted at 2017-08-19 06:33:33 on dist-test-slave-dist-test-slave-j33z" I0819 06:33:33.316462 1162 fs_report.cc:345] Block manager report -------------------- 1 data directories: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0819 06:33:33.499879 1162 master_main.cc:70] Starting Master server... I0819 06:33:33.501801 1179 process_memory.cc:172] Process hard memory limit is 11.777661 GB I0819 06:33:33.501951 1179 process_memory.cc:174] Process soft memory limit is 9.422129 GB I0819 06:33:33.502035 1179 process_memory.cc:177] Process memory pressure threshold is 7.066597 GB I0819 06:33:33.556433 1162 rpc_server.cc:186] RPC server started. Bound to: 127.4.126.254:34937 I0819 06:33:33.598747 1162 webserver.cc:145] Starting webserver on 127.4.126.254:0 I0819 06:33:33.598866 1162 webserver.cc:156] Document root disabled I0819 06:33:33.600826 1162 webserver.cc:278] Webserver started. Bound to: http://127.4.126.254:52430/ I0819 06:33:33.601651 1162 server_base.cc:359] Dumped server information to /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data/info.pb I0819 06:33:33.602658 1150 external_mini_cluster.cc:814] Started /tmp/run_tha_test479zHR/build/tsan/bin/kudu-master as pid 1162 I0819 06:33:33.604616 1150 external_mini_cluster.cc:752] Running /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data --rpc_bind_addresses=127.4.126.1:0 --local_ip_for_outbound_sockets=127.4.126.1 --webserver_interface=127.4.126.1 --webserver_port=0 --tserver_master_addrs=127.4.126.254:34937 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=flag --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=50 with env {} I0819 06:33:33.620030 1223 tablet_bootstrap.cc:402] T 00000000000000000000000000000000 P ff69923877524a91b05a9c6806bbac2f: Bootstrap starting. I0819 06:33:33.622436 1223 tablet_bootstrap.cc:537] T 00000000000000000000000000000000 P ff69923877524a91b05a9c6806bbac2f: No blocks or log segments found. Creating new log. I0819 06:33:33.623679 1223 log.cc:511] T 00000000000000000000000000000000 P ff69923877524a91b05a9c6806bbac2f: Log is configured to *not* fsync() on all Append() calls I0819 06:33:33.626093 1223 tablet_bootstrap.cc:402] T 00000000000000000000000000000000 P ff69923877524a91b05a9c6806bbac2f: No bootstrap required, opened a new log I0819 06:33:33.629114 1223 raft_consensus.cc:286] T 00000000000000000000000000000000 P ff69923877524a91b05a9c6806bbac2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff69923877524a91b05a9c6806bbac2f" member_type: VOTER } I0819 06:33:33.629619 1223 raft_consensus.cc:318] T 00000000000000000000000000000000 P ff69923877524a91b05a9c6806bbac2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0819 06:33:33.629773 1223 raft_consensus.cc:574] T 00000000000000000000000000000000 P ff69923877524a91b05a9c6806bbac2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff69923877524a91b05a9c6806bbac2f, State: Running, Role: FOLLOWER I0819 06:33:33.629963 1223 consensus_queue.cc:190] T 00000000000000000000000000000000 P ff69923877524a91b05a9c6806bbac2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0819 06:33:33.630163 1223 raft_consensus.cc:329] T 00000000000000000000000000000000 P ff69923877524a91b05a9c6806bbac2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0819 06:33:33.630296 1223 raft_consensus.cc:413] T 00000000000000000000000000000000 P ff69923877524a91b05a9c6806bbac2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0819 06:33:33.630434 1223 raft_consensus.cc:2401] T 00000000000000000000000000000000 P ff69923877524a91b05a9c6806bbac2f: Snoozing failure detection for election timeout plus an additional 2.097s I0819 06:33:33.630559 1223 raft_consensus.cc:2448] T 00000000000000000000000000000000 P ff69923877524a91b05a9c6806bbac2f [term 0 FOLLOWER]: Advancing to term 1 I0819 06:33:33.634073 1223 raft_consensus.cc:437] T 00000000000000000000000000000000 P ff69923877524a91b05a9c6806bbac2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff69923877524a91b05a9c6806bbac2f" member_type: VOTER } I0819 06:33:33.634563 1223 leader_election.cc:244] T 00000000000000000000000000000000 P ff69923877524a91b05a9c6806bbac2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0819 06:33:33.637552 1227 raft_consensus.cc:2401] T 00000000000000000000000000000000 P ff69923877524a91b05a9c6806bbac2f: Snoozing failure detection for election timeout plus an additional 1.531s I0819 06:33:33.637761 1227 raft_consensus.cc:2205] T 00000000000000000000000000000000 P ff69923877524a91b05a9c6806bbac2f [term 1 FOLLOWER]: Leader election won for term 1 I0819 06:33:33.653997 1227 raft_consensus.cc:540] T 00000000000000000000000000000000 P ff69923877524a91b05a9c6806bbac2f [term 1 LEADER]: Becoming Leader. State: Replica: ff69923877524a91b05a9c6806bbac2f, State: Running, Role: LEADER I0819 06:33:33.654340 1227 consensus_queue.cc:169] T 00000000000000000000000000000000 P ff69923877524a91b05a9c6806bbac2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff69923877524a91b05a9c6806bbac2f" member_type: VOTER } I0819 06:33:33.671918 1223 sys_catalog.cc:287] T 00000000000000000000000000000000 P ff69923877524a91b05a9c6806bbac2f [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "ff69923877524a91b05a9c6806bbac2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff69923877524a91b05a9c6806bbac2f" member_type: VOTER } } I0819 06:33:33.672349 1223 sys_catalog.cc:290] T 00000000000000000000000000000000 P ff69923877524a91b05a9c6806bbac2f [sys.catalog]: This master's current role is: LEADER I0819 06:33:33.672230 1228 sys_catalog.cc:287] T 00000000000000000000000000000000 P ff69923877524a91b05a9c6806bbac2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff69923877524a91b05a9c6806bbac2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff69923877524a91b05a9c6806bbac2f" member_type: VOTER } } I0819 06:33:33.673483 1228 sys_catalog.cc:290] T 00000000000000000000000000000000 P ff69923877524a91b05a9c6806bbac2f [sys.catalog]: This master's current role is: LEADER I0819 06:33:33.672385 1230 sys_catalog.cc:287] T 00000000000000000000000000000000 P ff69923877524a91b05a9c6806bbac2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff69923877524a91b05a9c6806bbac2f. Latest consensus state: current_term: 1 leader_uuid: "ff69923877524a91b05a9c6806bbac2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff69923877524a91b05a9c6806bbac2f" member_type: VOTER } } I0819 06:33:33.674018 1230 sys_catalog.cc:290] T 00000000000000000000000000000000 P ff69923877524a91b05a9c6806bbac2f [sys.catalog]: This master's current role is: LEADER I0819 06:33:33.676122 1223 sys_catalog.cc:369] T 00000000000000000000000000000000 P ff69923877524a91b05a9c6806bbac2f [sys.catalog]: configured and running, proceeding with master startup. I0819 06:33:33.676283 1232 catalog_manager.cc:1018] Loading table and tablet metadata into memory... I0819 06:33:33.685833 1162 master_main.cc:73] Master server successfully started. I0819 06:33:33.690873 1232 catalog_manager.cc:1029] Initializing Kudu internal certificate authority... I0819 06:33:33.735790 1232 catalog_manager.cc:914] Generated new certificate authority record I0819 06:33:33.737669 1232 catalog_manager.cc:1038] Loading token signing keys... I0819 06:33:33.749686 1232 catalog_manager.cc:3605] T 00000000000000000000000000000000 P ff69923877524a91b05a9c6806bbac2f: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0819 06:33:33.755339 1224 flags.cc:384] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0819 06:33:33.755735 1224 flags.cc:384] Enabled unsafe flag: --never_fsync=true W0819 06:33:33.773102 1224 flags.cc:384] Enabled experimental flag: --ipki_server_key_size=1024 W0819 06:33:33.773447 1224 flags.cc:384] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.126.1 I0819 06:33:33.807133 1224 tablet_server_main.cc:64] Tablet server non-default flags: --consensus_rpc_timeout_ms=50 --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.126.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.4.126.1 --webserver_port=0 --tserver_master_addrs=127.4.126.254:34937 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.1224 --redact=flag --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.126.1 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.5.0-SNAPSHOT revision ba0f21e2316b78f167bbaba44cab392558d97b6f build type FASTDEBUG built by None at 19 Aug 2017 06:31:02 UTC on jenkins-slave-c3oe build id 9422 TSAN enabled I0819 06:33:33.840950 1224 tablet_server_main.cc:71] Initializing tablet server... I0819 06:33:33.843497 1224 system_ntp.cc:96] NTP initialized. Resolution in nanos?: 1 Skew PPM: 500 Current error: 1280905 I0819 06:33:33.844164 1224 env_posix.cc:1586] Not raising process file limit of 65536; it is already as high as it can go I0819 06:33:33.844462 1224 file_cache.cc:464] Constructed file cache lbm with capacity 26214 I0819 06:33:33.849499 1224 server_base.cc:215] Could not load existing FS layout: Not found: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0819 06:33:33.849647 1224 server_base.cc:216] Creating new FS layout I0819 06:33:33.857319 1224 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "4c4e3833f6f34196b0a438f2bb35e27a" format_stamp: "Formatted at 2017-08-19 06:33:33 on dist-test-slave-dist-test-slave-j33z" I0819 06:33:33.858072 1224 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "4c4e3833f6f34196b0a438f2bb35e27a" format_stamp: "Formatted at 2017-08-19 06:33:33 on dist-test-slave-dist-test-slave-j33z" I0819 06:33:33.889643 1224 fs_manager.cc:280] Time spent opening block manager: real 0.025s user 0.004s sys 0.020s I0819 06:33:33.889798 1224 fs_manager.cc:283] Opened local filesystem: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data,/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/wal uuid: "4c4e3833f6f34196b0a438f2bb35e27a" format_stamp: "Formatted at 2017-08-19 06:33:33 on dist-test-slave-dist-test-slave-j33z" I0819 06:33:33.890019 1224 fs_report.cc:345] Block manager report -------------------- 1 data directories: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0819 06:33:34.194073 1224 ts_tablet_manager.cc:190] Loaded tablet metadata (0 live tablets) I0819 06:33:34.204880 1224 tablet_server_main.cc:76] Starting tablet server... I0819 06:33:34.594046 1224 rpc_server.cc:186] RPC server started. Bound to: 127.4.126.1:51099 I0819 06:33:34.594946 1224 webserver.cc:145] Starting webserver on 127.4.126.1:0 I0819 06:33:34.595049 1224 webserver.cc:156] Document root disabled I0819 06:33:34.596284 1224 webserver.cc:278] Webserver started. Bound to: http://127.4.126.1:34486/ I0819 06:33:34.597048 1224 server_base.cc:359] Dumped server information to /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/info.pb I0819 06:33:34.604898 1150 external_mini_cluster.cc:814] Started /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver as pid 1224 I0819 06:33:34.605769 1150 external_mini_cluster.cc:752] Running /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data --rpc_bind_addresses=127.4.126.2:0 --local_ip_for_outbound_sockets=127.4.126.2 --webserver_interface=127.4.126.2 --webserver_port=0 --tserver_master_addrs=127.4.126.254:34937 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=flag --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=50 with env {} I0819 06:33:34.604441 1224 tablet_server_main.cc:79] Tablet server successfully started. I0819 06:33:34.628481 1356 heartbeater.cc:295] Connected to a master server at 127.4.126.254:34937 I0819 06:33:34.628760 1356 heartbeater.cc:365] Registering TS with master... I0819 06:33:34.629530 1356 heartbeater.cc:392] Master 127.4.126.254:34937 requested a full tablet report, sending... I0819 06:33:34.631259 1190 ts_manager.cc:79] Registered new tserver with Master: 4c4e3833f6f34196b0a438f2bb35e27a (127.4.126.1:51099) I0819 06:33:34.632825 1190 master_service.cc:197] Signed X509 certificate for tserver {username='slave'} at 127.4.126.1:57983 I0819 06:33:34.608714 1357 process_memory.cc:172] Process hard memory limit is 11.777661 GB I0819 06:33:34.637886 1357 process_memory.cc:174] Process soft memory limit is 9.422129 GB I0819 06:33:34.637972 1357 process_memory.cc:177] Process memory pressure threshold is 7.066597 GB WARNING: Logging before InitGoogleLogging() is written to STDERR W0819 06:33:34.740597 1360 flags.cc:384] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0819 06:33:34.741001 1360 flags.cc:384] Enabled unsafe flag: --never_fsync=true W0819 06:33:34.756944 1360 flags.cc:384] Enabled experimental flag: --ipki_server_key_size=1024 W0819 06:33:34.757295 1360 flags.cc:384] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.126.2 I0819 06:33:34.779161 1360 tablet_server_main.cc:64] Tablet server non-default flags: --consensus_rpc_timeout_ms=50 --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.126.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.4.126.2 --webserver_port=0 --tserver_master_addrs=127.4.126.254:34937 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.1360 --redact=flag --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.126.2 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.5.0-SNAPSHOT revision ba0f21e2316b78f167bbaba44cab392558d97b6f build type FASTDEBUG built by None at 19 Aug 2017 06:31:02 UTC on jenkins-slave-c3oe build id 9422 TSAN enabled I0819 06:33:34.791177 1360 tablet_server_main.cc:71] Initializing tablet server... I0819 06:33:34.793149 1360 system_ntp.cc:96] NTP initialized. Resolution in nanos?: 1 Skew PPM: 500 Current error: 1281405 I0819 06:33:34.793817 1360 env_posix.cc:1586] Not raising process file limit of 65536; it is already as high as it can go I0819 06:33:34.794095 1360 file_cache.cc:464] Constructed file cache lbm with capacity 26214 I0819 06:33:34.799175 1360 server_base.cc:215] Could not load existing FS layout: Not found: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0819 06:33:34.799324 1360 server_base.cc:216] Creating new FS layout I0819 06:33:34.806955 1360 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "1aaa61f0808e4d84877f034e1b68a402" format_stamp: "Formatted at 2017-08-19 06:33:34 on dist-test-slave-dist-test-slave-j33z" I0819 06:33:34.807713 1360 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "1aaa61f0808e4d84877f034e1b68a402" format_stamp: "Formatted at 2017-08-19 06:33:34 on dist-test-slave-dist-test-slave-j33z" I0819 06:33:34.819223 1360 fs_manager.cc:280] Time spent opening block manager: real 0.005s user 0.008s sys 0.000s I0819 06:33:34.819375 1360 fs_manager.cc:283] Opened local filesystem: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data,/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/wal uuid: "1aaa61f0808e4d84877f034e1b68a402" format_stamp: "Formatted at 2017-08-19 06:33:34 on dist-test-slave-dist-test-slave-j33z" I0819 06:33:34.819577 1360 fs_report.cc:345] Block manager report -------------------- 1 data directories: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0819 06:33:35.010551 1360 ts_tablet_manager.cc:190] Loaded tablet metadata (0 live tablets) I0819 06:33:35.012676 1360 tablet_server_main.cc:76] Starting tablet server... I0819 06:33:35.133379 1360 rpc_server.cc:186] RPC server started. Bound to: 127.4.126.2:50325 I0819 06:33:35.134259 1360 webserver.cc:145] Starting webserver on 127.4.126.2:0 I0819 06:33:35.134364 1360 webserver.cc:156] Document root disabled I0819 06:33:35.135553 1360 webserver.cc:278] Webserver started. Bound to: http://127.4.126.2:58347/ I0819 06:33:35.136299 1360 server_base.cc:359] Dumped server information to /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/info.pb I0819 06:33:35.138950 1150 external_mini_cluster.cc:814] Started /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver as pid 1360 I0819 06:33:35.139832 1150 external_mini_cluster.cc:752] Running /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data --rpc_bind_addresses=127.4.126.3:0 --local_ip_for_outbound_sockets=127.4.126.3 --webserver_interface=127.4.126.3 --webserver_port=0 --tserver_master_addrs=127.4.126.254:34937 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=flag --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=50 with env {} I0819 06:33:35.139150 1482 process_memory.cc:172] Process hard memory limit is 11.777661 GB I0819 06:33:35.140185 1482 process_memory.cc:174] Process soft memory limit is 9.422129 GB I0819 06:33:35.140275 1482 process_memory.cc:177] Process memory pressure threshold is 7.066597 GB I0819 06:33:35.140130 1360 tablet_server_main.cc:79] Tablet server successfully started. I0819 06:33:35.202373 1481 heartbeater.cc:295] Connected to a master server at 127.4.126.254:34937 I0819 06:33:35.202666 1481 heartbeater.cc:365] Registering TS with master... I0819 06:33:35.203104 1481 heartbeater.cc:392] Master 127.4.126.254:34937 requested a full tablet report, sending... I0819 06:33:35.204535 1190 ts_manager.cc:79] Registered new tserver with Master: 1aaa61f0808e4d84877f034e1b68a402 (127.4.126.2:50325) I0819 06:33:35.206172 1190 master_service.cc:197] Signed X509 certificate for tserver {username='slave'} at 127.4.126.2:50730 WARNING: Logging before InitGoogleLogging() is written to STDERR W0819 06:33:35.284809 1483 flags.cc:384] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0819 06:33:35.285203 1483 flags.cc:384] Enabled unsafe flag: --never_fsync=true W0819 06:33:35.300824 1483 flags.cc:384] Enabled experimental flag: --ipki_server_key_size=1024 W0819 06:33:35.301158 1483 flags.cc:384] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.126.3 I0819 06:33:35.322415 1483 tablet_server_main.cc:64] Tablet server non-default flags: --consensus_rpc_timeout_ms=50 --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.126.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.4.126.3 --webserver_port=0 --tserver_master_addrs=127.4.126.254:34937 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.1483 --redact=flag --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.126.3 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.5.0-SNAPSHOT revision ba0f21e2316b78f167bbaba44cab392558d97b6f build type FASTDEBUG built by None at 19 Aug 2017 06:31:02 UTC on jenkins-slave-c3oe build id 9422 TSAN enabled I0819 06:33:35.333353 1483 tablet_server_main.cc:71] Initializing tablet server... I0819 06:33:35.335793 1483 system_ntp.cc:96] NTP initialized. Resolution in nanos?: 1 Skew PPM: 500 Current error: 1281905 I0819 06:33:35.336432 1483 env_posix.cc:1586] Not raising process file limit of 65536; it is already as high as it can go I0819 06:33:35.336736 1483 file_cache.cc:464] Constructed file cache lbm with capacity 26214 I0819 06:33:35.341876 1483 server_base.cc:215] Could not load existing FS layout: Not found: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0819 06:33:35.342041 1483 server_base.cc:216] Creating new FS layout I0819 06:33:35.349669 1483 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "a4f09dfa55a941379c23044b0fad5ab3" format_stamp: "Formatted at 2017-08-19 06:33:35 on dist-test-slave-dist-test-slave-j33z" I0819 06:33:35.350404 1483 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "a4f09dfa55a941379c23044b0fad5ab3" format_stamp: "Formatted at 2017-08-19 06:33:35 on dist-test-slave-dist-test-slave-j33z" I0819 06:33:35.361963 1483 fs_manager.cc:280] Time spent opening block manager: real 0.005s user 0.000s sys 0.000s I0819 06:33:35.362108 1483 fs_manager.cc:283] Opened local filesystem: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data,/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/wal uuid: "a4f09dfa55a941379c23044b0fad5ab3" format_stamp: "Formatted at 2017-08-19 06:33:35 on dist-test-slave-dist-test-slave-j33z" I0819 06:33:35.362293 1483 fs_report.cc:345] Block manager report -------------------- 1 data directories: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0819 06:33:35.581291 1483 ts_tablet_manager.cc:190] Loaded tablet metadata (0 live tablets) I0819 06:33:35.582928 1483 tablet_server_main.cc:76] Starting tablet server... I0819 06:33:35.635556 1356 heartbeater.cc:384] Master 127.4.126.254:34937 was elected leader, sending a full tablet report... I0819 06:33:35.705669 1483 rpc_server.cc:186] RPC server started. Bound to: 127.4.126.3:56715 I0819 06:33:35.706567 1483 webserver.cc:145] Starting webserver on 127.4.126.3:0 I0819 06:33:35.706671 1483 webserver.cc:156] Document root disabled I0819 06:33:35.708552 1483 webserver.cc:278] Webserver started. Bound to: http://127.4.126.3:48936/ I0819 06:33:35.709306 1483 server_base.cc:359] Dumped server information to /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/info.pb I0819 06:33:35.716928 1150 external_mini_cluster.cc:814] Started /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver as pid 1483 I0819 06:33:35.719336 1483 tablet_server_main.cc:79] Tablet server successfully started. I0819 06:33:35.720130 1607 process_memory.cc:172] Process hard memory limit is 11.777661 GB I0819 06:33:35.744437 1607 process_memory.cc:174] Process soft memory limit is 9.422129 GB I0819 06:33:35.744565 1607 process_memory.cc:177] Process memory pressure threshold is 7.066597 GB I0819 06:33:35.748580 1606 heartbeater.cc:295] Connected to a master server at 127.4.126.254:34937 I0819 06:33:35.748859 1606 heartbeater.cc:365] Registering TS with master... I0819 06:33:35.749275 1606 heartbeater.cc:392] Master 127.4.126.254:34937 requested a full tablet report, sending... I0819 06:33:35.750741 1190 ts_manager.cc:79] Registered new tserver with Master: a4f09dfa55a941379c23044b0fad5ab3 (127.4.126.3:56715) I0819 06:33:35.751809 1190 master_service.cc:197] Signed X509 certificate for tserver {username='slave'} at 127.4.126.3:36722 I0819 06:33:35.755877 1150 external_mini_cluster.cc:454] 3 TS(s) registered with all masters I0819 06:33:35.786440 1190 catalog_manager.cc:1281] CreateTable from {username='slave'} at 127.0.0.1:48886: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0819 06:33:35.837345 1293 tablet_service.cc:684] Processing CreateTablet for tablet 46b81f03fcaa4a0093a062ad958e8a35 (table=TestTable [id=c5203443b0804dc19ef01fda7db1460d]), partition=RANGE (key) PARTITION UNBOUNDED I0819 06:33:35.837673 1543 tablet_service.cc:684] Processing CreateTablet for tablet 46b81f03fcaa4a0093a062ad958e8a35 (table=TestTable [id=c5203443b0804dc19ef01fda7db1460d]), partition=RANGE (key) PARTITION UNBOUNDED I0819 06:33:35.839777 1418 tablet_service.cc:684] Processing CreateTablet for tablet 46b81f03fcaa4a0093a062ad958e8a35 (table=TestTable [id=c5203443b0804dc19ef01fda7db1460d]), partition=RANGE (key) PARTITION UNBOUNDED I0819 06:33:35.852280 1293 ts_tablet_manager.cc:877] T 46b81f03fcaa4a0093a062ad958e8a35 P 4c4e3833f6f34196b0a438f2bb35e27a: Registered tablet (data state: TABLET_DATA_READY) I0819 06:33:35.858857 1543 ts_tablet_manager.cc:877] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3: Registered tablet (data state: TABLET_DATA_READY) I0819 06:33:35.860309 1418 ts_tablet_manager.cc:877] T 46b81f03fcaa4a0093a062ad958e8a35 P 1aaa61f0808e4d84877f034e1b68a402: Registered tablet (data state: TABLET_DATA_READY) I0819 06:33:35.864560 1627 ts_tablet_manager.cc:740] T 46b81f03fcaa4a0093a062ad958e8a35 P 4c4e3833f6f34196b0a438f2bb35e27a: Bootstrapping tablet I0819 06:33:35.864989 1627 tablet_bootstrap.cc:402] T 46b81f03fcaa4a0093a062ad958e8a35 P 4c4e3833f6f34196b0a438f2bb35e27a: Bootstrap starting. I0819 06:33:35.867331 1627 tablet_bootstrap.cc:537] T 46b81f03fcaa4a0093a062ad958e8a35 P 4c4e3833f6f34196b0a438f2bb35e27a: No blocks or log segments found. Creating new log. I0819 06:33:35.869220 1627 log.cc:511] T 46b81f03fcaa4a0093a062ad958e8a35 P 4c4e3833f6f34196b0a438f2bb35e27a: Log is configured to *not* fsync() on all Append() calls I0819 06:33:35.872231 1629 ts_tablet_manager.cc:740] T 46b81f03fcaa4a0093a062ad958e8a35 P 1aaa61f0808e4d84877f034e1b68a402: Bootstrapping tablet I0819 06:33:35.872730 1629 tablet_bootstrap.cc:402] T 46b81f03fcaa4a0093a062ad958e8a35 P 1aaa61f0808e4d84877f034e1b68a402: Bootstrap starting. I0819 06:33:35.873425 1627 tablet_bootstrap.cc:402] T 46b81f03fcaa4a0093a062ad958e8a35 P 4c4e3833f6f34196b0a438f2bb35e27a: No bootstrap required, opened a new log I0819 06:33:35.873461 1628 ts_tablet_manager.cc:740] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3: Bootstrapping tablet I0819 06:33:35.873878 1627 ts_tablet_manager.cc:752] T 46b81f03fcaa4a0093a062ad958e8a35 P 4c4e3833f6f34196b0a438f2bb35e27a: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s I0819 06:33:35.873924 1628 tablet_bootstrap.cc:402] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3: Bootstrap starting. I0819 06:33:35.877822 1629 tablet_bootstrap.cc:537] T 46b81f03fcaa4a0093a062ad958e8a35 P 1aaa61f0808e4d84877f034e1b68a402: No blocks or log segments found. Creating new log. I0819 06:33:35.879762 1628 tablet_bootstrap.cc:537] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3: No blocks or log segments found. Creating new log. I0819 06:33:35.880465 1629 log.cc:511] T 46b81f03fcaa4a0093a062ad958e8a35 P 1aaa61f0808e4d84877f034e1b68a402: Log is configured to *not* fsync() on all Append() calls I0819 06:33:35.880131 1627 raft_consensus.cc:286] T 46b81f03fcaa4a0093a062ad958e8a35 P 4c4e3833f6f34196b0a438f2bb35e27a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f09dfa55a941379c23044b0fad5ab3" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 56715 } } peers { permanent_uuid: "1aaa61f0808e4d84877f034e1b68a402" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 50325 } } peers { permanent_uuid: "4c4e3833f6f34196b0a438f2bb35e27a" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 51099 } } I0819 06:33:35.881381 1627 raft_consensus.cc:318] T 46b81f03fcaa4a0093a062ad958e8a35 P 4c4e3833f6f34196b0a438f2bb35e27a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0819 06:33:35.881614 1627 raft_consensus.cc:574] T 46b81f03fcaa4a0093a062ad958e8a35 P 4c4e3833f6f34196b0a438f2bb35e27a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c4e3833f6f34196b0a438f2bb35e27a, State: Running, Role: FOLLOWER I0819 06:33:35.881974 1627 consensus_queue.cc:190] T 46b81f03fcaa4a0093a062ad958e8a35 P 4c4e3833f6f34196b0a438f2bb35e27a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0819 06:33:35.882243 1628 log.cc:511] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3: Log is configured to *not* fsync() on all Append() calls I0819 06:33:35.885871 1627 ts_tablet_manager.cc:781] T 46b81f03fcaa4a0093a062ad958e8a35 P 4c4e3833f6f34196b0a438f2bb35e27a: Time spent starting tablet: real 0.012s user 0.004s sys 0.008s I0819 06:33:35.886549 1628 tablet_bootstrap.cc:402] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3: No bootstrap required, opened a new log I0819 06:33:35.886945 1628 ts_tablet_manager.cc:752] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s I0819 06:33:35.888865 1629 tablet_bootstrap.cc:402] T 46b81f03fcaa4a0093a062ad958e8a35 P 1aaa61f0808e4d84877f034e1b68a402: No bootstrap required, opened a new log I0819 06:33:35.889250 1629 ts_tablet_manager.cc:752] T 46b81f03fcaa4a0093a062ad958e8a35 P 1aaa61f0808e4d84877f034e1b68a402: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.008s I0819 06:33:35.893435 1628 raft_consensus.cc:286] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f09dfa55a941379c23044b0fad5ab3" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 56715 } } peers { permanent_uuid: "1aaa61f0808e4d84877f034e1b68a402" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 50325 } } peers { permanent_uuid: "4c4e3833f6f34196b0a438f2bb35e27a" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 51099 } } I0819 06:33:35.893806 1629 raft_consensus.cc:286] T 46b81f03fcaa4a0093a062ad958e8a35 P 1aaa61f0808e4d84877f034e1b68a402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f09dfa55a941379c23044b0fad5ab3" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 56715 } } peers { permanent_uuid: "1aaa61f0808e4d84877f034e1b68a402" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 50325 } } peers { permanent_uuid: "4c4e3833f6f34196b0a438f2bb35e27a" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 51099 } } I0819 06:33:35.894449 1628 raft_consensus.cc:318] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0819 06:33:35.894668 1628 raft_consensus.cc:574] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4f09dfa55a941379c23044b0fad5ab3, State: Running, Role: FOLLOWER I0819 06:33:35.895565 1629 raft_consensus.cc:318] T 46b81f03fcaa4a0093a062ad958e8a35 P 1aaa61f0808e4d84877f034e1b68a402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0819 06:33:35.895546 1628 consensus_queue.cc:190] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0819 06:33:35.895766 1629 raft_consensus.cc:574] T 46b81f03fcaa4a0093a062ad958e8a35 P 1aaa61f0808e4d84877f034e1b68a402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aaa61f0808e4d84877f034e1b68a402, State: Running, Role: FOLLOWER I0819 06:33:35.896060 1629 consensus_queue.cc:190] T 46b81f03fcaa4a0093a062ad958e8a35 P 1aaa61f0808e4d84877f034e1b68a402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0819 06:33:35.897915 1606 heartbeater.cc:384] Master 127.4.126.254:34937 was elected leader, sending a full tablet report... I0819 06:33:35.898646 1481 heartbeater.cc:384] Master 127.4.126.254:34937 was elected leader, sending a full tablet report... I0819 06:33:35.899257 1628 ts_tablet_manager.cc:781] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3: Time spent starting tablet: real 0.012s user 0.004s sys 0.008s I0819 06:33:35.899772 1629 ts_tablet_manager.cc:781] T 46b81f03fcaa4a0093a062ad958e8a35 P 1aaa61f0808e4d84877f034e1b68a402: Time spent starting tablet: real 0.010s user 0.004s sys 0.004s I0819 06:33:36.039630 1635 raft_consensus.cc:413] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0819 06:33:36.039887 1635 raft_consensus.cc:2401] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3: Snoozing failure detection for election timeout plus an additional 1.681s I0819 06:33:36.040081 1635 raft_consensus.cc:437] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f09dfa55a941379c23044b0fad5ab3" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 56715 } } peers { permanent_uuid: "1aaa61f0808e4d84877f034e1b68a402" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 50325 } } peers { permanent_uuid: "4c4e3833f6f34196b0a438f2bb35e27a" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 51099 } } I0819 06:33:36.041725 1635 leader_election.cc:217] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1aaa61f0808e4d84877f034e1b68a402 I0819 06:33:36.042222 1635 leader_election.cc:217] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 4c4e3833f6f34196b0a438f2bb35e27a I0819 06:33:36.058984 1438 raft_consensus.cc:2401] T 46b81f03fcaa4a0093a062ad958e8a35 P 1aaa61f0808e4d84877f034e1b68a402: Snoozing failure detection for election timeout plus an additional 1.753s I0819 06:33:36.059108 1313 raft_consensus.cc:2401] T 46b81f03fcaa4a0093a062ad958e8a35 P 4c4e3833f6f34196b0a438f2bb35e27a: Snoozing failure detection for election timeout plus an additional 2.198s I0819 06:33:36.059223 1438 raft_consensus.cc:1967] T 46b81f03fcaa4a0093a062ad958e8a35 P 1aaa61f0808e4d84877f034e1b68a402 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4f09dfa55a941379c23044b0fad5ab3 in term 0. I0819 06:33:36.059345 1313 raft_consensus.cc:1967] T 46b81f03fcaa4a0093a062ad958e8a35 P 4c4e3833f6f34196b0a438f2bb35e27a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4f09dfa55a941379c23044b0fad5ab3 in term 0. I0819 06:33:36.060191 1495 leader_election.cc:349] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1aaa61f0808e4d84877f034e1b68a402 I0819 06:33:36.060384 1495 leader_election.cc:244] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0819 06:33:36.061280 1496 leader_election.cc:349] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 4c4e3833f6f34196b0a438f2bb35e27a I0819 06:33:36.061529 1637 raft_consensus.cc:2401] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3: Snoozing failure detection for election timeout plus an additional 1.813s I0819 06:33:36.061743 1637 raft_consensus.cc:2205] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0819 06:33:36.061889 1637 raft_consensus.cc:413] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0819 06:33:36.062017 1637 raft_consensus.cc:2401] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3: Snoozing failure detection for election timeout plus an additional 1.785s I0819 06:33:36.062124 1637 raft_consensus.cc:2448] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3 [term 0 FOLLOWER]: Advancing to term 1 I0819 06:33:36.064874 1637 raft_consensus.cc:437] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f09dfa55a941379c23044b0fad5ab3" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 56715 } } peers { permanent_uuid: "1aaa61f0808e4d84877f034e1b68a402" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 50325 } } peers { permanent_uuid: "4c4e3833f6f34196b0a438f2bb35e27a" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 51099 } } I0819 06:33:36.065629 1637 leader_election.cc:217] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3 [CANDIDATE]: Term 1 election: Requesting vote from peer 1aaa61f0808e4d84877f034e1b68a402 I0819 06:33:36.065954 1637 leader_election.cc:217] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3 [CANDIDATE]: Term 1 election: Requesting vote from peer 4c4e3833f6f34196b0a438f2bb35e27a I0819 06:33:36.067009 1438 raft_consensus.cc:2448] T 46b81f03fcaa4a0093a062ad958e8a35 P 1aaa61f0808e4d84877f034e1b68a402 [term 0 FOLLOWER]: Advancing to term 1 I0819 06:33:36.067319 1438 raft_consensus.cc:2401] T 46b81f03fcaa4a0093a062ad958e8a35 P 1aaa61f0808e4d84877f034e1b68a402: Snoozing failure detection for election timeout plus an additional 2.232s I0819 06:33:36.067836 1313 raft_consensus.cc:2448] T 46b81f03fcaa4a0093a062ad958e8a35 P 4c4e3833f6f34196b0a438f2bb35e27a [term 0 FOLLOWER]: Advancing to term 1 I0819 06:33:36.068132 1313 raft_consensus.cc:2401] T 46b81f03fcaa4a0093a062ad958e8a35 P 4c4e3833f6f34196b0a438f2bb35e27a: Snoozing failure detection for election timeout plus an additional 1.753s I0819 06:33:36.069946 1438 raft_consensus.cc:1967] T 46b81f03fcaa4a0093a062ad958e8a35 P 1aaa61f0808e4d84877f034e1b68a402 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4f09dfa55a941379c23044b0fad5ab3 in term 1. I0819 06:33:36.070801 1313 raft_consensus.cc:1967] T 46b81f03fcaa4a0093a062ad958e8a35 P 4c4e3833f6f34196b0a438f2bb35e27a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4f09dfa55a941379c23044b0fad5ab3 in term 1. I0819 06:33:36.071367 1495 leader_election.cc:349] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3 [CANDIDATE]: Term 1 election: Vote granted by peer 1aaa61f0808e4d84877f034e1b68a402 I0819 06:33:36.071528 1495 leader_election.cc:244] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0819 06:33:36.071701 1496 leader_election.cc:349] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3 [CANDIDATE]: Term 1 election: Vote granted by peer 4c4e3833f6f34196b0a438f2bb35e27a I0819 06:33:36.071857 1637 raft_consensus.cc:2401] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3: Snoozing failure detection for election timeout plus an additional 1.633s I0819 06:33:36.072027 1637 raft_consensus.cc:2205] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3 [term 1 FOLLOWER]: Leader election won for term 1 I0819 06:33:36.073923 1637 raft_consensus.cc:540] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3 [term 1 LEADER]: Becoming Leader. State: Replica: a4f09dfa55a941379c23044b0fad5ab3, State: Running, Role: LEADER I0819 06:33:36.074259 1637 consensus_queue.cc:169] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f09dfa55a941379c23044b0fad5ab3" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 56715 } } peers { permanent_uuid: "1aaa61f0808e4d84877f034e1b68a402" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 50325 } } peers { permanent_uuid: "4c4e3833f6f34196b0a438f2bb35e27a" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 51099 } } I0819 06:33:36.089084 1190 catalog_manager.cc:2559] T 46b81f03fcaa4a0093a062ad958e8a35 reported consensus state change: term changed from 0 to 1, leader changed from to a4f09dfa55a941379c23044b0fad5ab3 (127.4.126.3). New consensus state: current_term: 1 leader_uuid: "a4f09dfa55a941379c23044b0fad5ab3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f09dfa55a941379c23044b0fad5ab3" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 56715 } } peers { permanent_uuid: "1aaa61f0808e4d84877f034e1b68a402" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 50325 } } peers { permanent_uuid: "4c4e3833f6f34196b0a438f2bb35e27a" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 51099 } } } I0819 06:33:36.128278 1150 external_mini_cluster.cc:454] 3 TS(s) registered with all masters I0819 06:33:36.131374 1150 ts_itest-base.h:198] Waiting for 1 tablets on tserver 4c4e3833f6f34196b0a438f2bb35e27a to finish bootstrapping I0819 06:33:36.140811 1150 ts_itest-base.h:198] Waiting for 1 tablets on tserver 1aaa61f0808e4d84877f034e1b68a402 to finish bootstrapping I0819 06:33:36.150609 1150 ts_itest-base.h:198] Waiting for 1 tablets on tserver a4f09dfa55a941379c23044b0fad5ab3 to finish bootstrapping I0819 06:33:36.188405 1150 cluster_itest_util.cc:220] Not converged past 1 yet: 0.0 0.0 1.1 I0819 06:33:36.293213 1150 cluster_itest_util.cc:220] Not converged past 1 yet: 0.0 0.0 1.1 I0819 06:33:36.498545 1150 cluster_itest_util.cc:220] Not converged past 1 yet: 0.0 0.0 1.1 I0819 06:33:36.649181 1313 raft_consensus.cc:961] T 46b81f03fcaa4a0093a062ad958e8a35 P 4c4e3833f6f34196b0a438f2bb35e27a [term 1 FOLLOWER]: Refusing update from remote peer a4f09dfa55a941379c23044b0fad5ab3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0819 06:33:36.651108 1643 consensus_queue.cc:726] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3 [LEADER]: Connected to new peer: Peer: 4c4e3833f6f34196b0a438f2bb35e27a, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I0819 06:33:36.667109 1438 raft_consensus.cc:961] T 46b81f03fcaa4a0093a062ad958e8a35 P 1aaa61f0808e4d84877f034e1b68a402 [term 1 FOLLOWER]: Refusing update from remote peer a4f09dfa55a941379c23044b0fad5ab3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0819 06:33:36.670864 1646 consensus_queue.cc:726] T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3 [LEADER]: Connected to new peer: Peer: 1aaa61f0808e4d84877f034e1b68a402, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false Connected to the Master Fetched info from all 3 Tablet Servers Table TestTable is HEALTHY (1 tablet(s) checked) The metadata for 1 table(s) is HEALTHY Checksum finished in 0s: 0/3 replicas remaining (0B from disk, 0 rows summed) ----------------------- TestTable ----------------------- T 46b81f03fcaa4a0093a062ad958e8a35 P 4c4e3833f6f34196b0a438f2bb35e27a (127.4.126.1:51099): Checksum: 0 T 46b81f03fcaa4a0093a062ad958e8a35 P a4f09dfa55a941379c23044b0fad5ab3 (127.4.126.3:56715): Checksum: 0 T 46b81f03fcaa4a0093a062ad958e8a35 P 1aaa61f0808e4d84877f034e1b68a402 (127.4.126.2:50325): Checksum: 0 I0819 06:33:36.932994 1150 log_verifier.cc:125] Checking tablet 46b81f03fcaa4a0093a062ad958e8a35 I0819 06:33:36.936906 1150 env_posix.cc:1586] Not raising process file limit of 65536; it is already as high as it can go I0819 06:33:36.941130 1150 file_cache.cc:464] Constructed file cache lbm with capacity 26214 W0819 06:33:36.955348 1150 data_dirs.cc:364] IO error: Could not lock /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/data/block_manager_instance: Could not lock /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/data/block_manager_instance: lock /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/data/block_manager_instance: Resource temporarily unavailable (error 11) W0819 06:33:36.955703 1150 data_dirs.cc:365] Proceeding without lock I0819 06:33:36.985652 1678 log_block_manager.cc:2195] Read-only block manager, skipping repair I0819 06:33:36.987486 1150 fs_report.cc:345] Block manager report -------------------- 1 data directories: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0819 06:33:36.988178 1150 fs_manager.cc:280] Time spent opening block manager: real 0.034s user 0.012s sys 0.020s I0819 06:33:36.988422 1150 fs_manager.cc:283] Opened local filesystem: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data,/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/wal uuid: "4c4e3833f6f34196b0a438f2bb35e27a" format_stamp: "Formatted at 2017-08-19 06:33:33 on dist-test-slave-dist-test-slave-j33z" I0819 06:33:37.032548 1150 file_cache.cc:464] Constructed file cache lbm with capacity 26214 W0819 06:33:37.043709 1150 data_dirs.cc:364] IO error: Could not lock /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/data/block_manager_instance: Could not lock /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/data/block_manager_instance: lock /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/data/block_manager_instance: Resource temporarily unavailable (error 11) W0819 06:33:37.044006 1150 data_dirs.cc:365] Proceeding without lock I0819 06:33:37.089283 1680 log_block_manager.cc:2195] Read-only block manager, skipping repair I0819 06:33:37.089674 1150 fs_report.cc:345] Block manager report -------------------- 1 data directories: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0819 06:33:37.090188 1150 fs_manager.cc:280] Time spent opening block manager: real 0.047s user 0.016s sys 0.028s I0819 06:33:37.090374 1150 fs_manager.cc:283] Opened local filesystem: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data,/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/wal uuid: "1aaa61f0808e4d84877f034e1b68a402" format_stamp: "Formatted at 2017-08-19 06:33:34 on dist-test-slave-dist-test-slave-j33z" I0819 06:33:37.131515 1150 file_cache.cc:464] Constructed file cache lbm with capacity 26214 W0819 06:33:37.147100 1150 data_dirs.cc:364] IO error: Could not lock /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/data/block_manager_instance: Could not lock /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/data/block_manager_instance: lock /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/data/block_manager_instance: Resource temporarily unavailable (error 11) W0819 06:33:37.147393 1150 data_dirs.cc:365] Proceeding without lock I0819 06:33:37.168177 1682 log_block_manager.cc:2195] Read-only block manager, skipping repair I0819 06:33:37.168591 1150 fs_report.cc:345] Block manager report -------------------- 1 data directories: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0819 06:33:37.169090 1150 fs_manager.cc:280] Time spent opening block manager: real 0.023s user 0.000s sys 0.020s I0819 06:33:37.169277 1150 fs_manager.cc:283] Opened local filesystem: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data,/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestInsertOnNonLeader.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/wal uuid: "a4f09dfa55a941379c23044b0fad5ab3" format_stamp: "Formatted at 2017-08-19 06:33:35 on dist-test-slave-dist-test-slave-j33z" I0819 06:33:37.237205 1150 log_verifier.cc:178] Verified matching terms for 1 ops in tablet 46b81f03fcaa4a0093a062ad958e8a35 I0819 06:33:37.591472 1150 external_mini_cluster.cc:944] Killing /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver with pid 1224 W0819 06:33:37.640751 1496 connection.cc:551] client connection to 127.4.126.1:51099 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:33:37.640724 1173 connection.cc:551] client connection to 127.4.126.1:51099 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:33:37.641676 1171 connection.cc:551] server connection from 127.4.126.1:57983 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0819 06:33:37.642096 1150 external_mini_cluster.cc:944] Killing /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver with pid 1360 W0819 06:33:37.666096 1172 connection.cc:551] client connection to 127.4.126.2:50325 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:33:37.667089 1173 connection.cc:551] server connection from 127.4.126.2:50730 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:33:37.667783 1495 connection.cc:551] client connection to 127.4.126.2:50325 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0819 06:33:37.669301 1150 external_mini_cluster.cc:944] Killing /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver with pid 1483 W0819 06:33:37.642544 1158 connection.cc:551] client connection to 127.4.126.1:51099 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:33:37.699905 1175 connection.cc:551] client connection to 127.4.126.3:56715 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:33:37.700892 1171 connection.cc:551] server connection from 127.4.126.3:36722 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:33:37.701648 1160 connection.cc:551] client connection to 127.4.126.3:56715 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0819 06:33:37.702695 1150 external_mini_cluster.cc:944] Killing /tmp/run_tha_test479zHR/build/tsan/bin/kudu-master with pid 1162 W0819 06:33:37.643133 1161 connection.cc:551] client connection to 127.4.126.1:51099 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) [ OK ] RaftConsensusITest.TestInsertOnNonLeader (4830 ms) [ RUN ] RaftConsensusITest.TestChurnyElections_WithDuplicateKeys I0819 06:33:37.742028 1150 test_util.cc:185] Using random seed: 1828258993 W0819 06:33:37.668774 1157 connection.cc:551] client connection to 127.4.126.2:50325 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0819 06:33:37.776819 1150 ts_itest-base.h:81] Starting cluster with: I0819 06:33:37.776933 1150 ts_itest-base.h:82] -------------- I0819 06:33:37.777009 1150 ts_itest-base.h:83] 3 tablet servers I0819 06:33:37.777081 1150 ts_itest-base.h:84] 3 replicas per TS I0819 06:33:37.777153 1150 ts_itest-base.h:85] -------------- I0819 06:33:37.779307 1150 external_mini_cluster.cc:752] Running /tmp/run_tha_test479zHR/build/tsan/bin/kudu-master /tmp/run_tha_test479zHR/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.126.254:0 --webserver_interface=127.4.126.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=flag --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0819 06:33:37.907282 1695 flags.cc:384] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0819 06:33:37.907670 1695 flags.cc:384] Enabled unsafe flag: --never_fsync=true W0819 06:33:37.923357 1695 flags.cc:384] Enabled experimental flag: --ipki_ca_key_size=1024 W0819 06:33:37.923509 1695 flags.cc:384] Enabled experimental flag: --ipki_server_key_size=1024 W0819 06:33:37.923607 1695 flags.cc:384] Enabled experimental flag: --tsk_num_rsa_bits=512 I0819 06:33:37.944835 1695 master_main.cc:60] Master server non-default flags: --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.126.254:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.4.126.254 --webserver_port=0 --never_fsync=true --heap_profile_path=/tmp/kudu-master.1695 --redact=flag --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.5.0-SNAPSHOT revision ba0f21e2316b78f167bbaba44cab392558d97b6f build type FASTDEBUG built by None at 19 Aug 2017 06:31:02 UTC on jenkins-slave-c3oe build id 9422 TSAN enabled I0819 06:33:37.954565 1695 master_main.cc:67] Initializing master server... I0819 06:33:37.956181 1695 system_ntp.cc:96] NTP initialized. Resolution in nanos?: 1 Skew PPM: 500 Current error: 1282905 I0819 06:33:37.957137 1695 env_posix.cc:1586] Not raising process file limit of 65536; it is already as high as it can go I0819 06:33:37.957413 1695 file_cache.cc:464] Constructed file cache lbm with capacity 26214 I0819 06:33:37.962373 1695 server_base.cc:215] Could not load existing FS layout: Not found: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0819 06:33:37.962520 1695 server_base.cc:216] Creating new FS layout I0819 06:33:37.970037 1695 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data/instance: uuid: "4d439876eac04b058bc1946edf883aa4" format_stamp: "Formatted at 2017-08-19 06:33:37 on dist-test-slave-dist-test-slave-j33z" I0819 06:33:37.970837 1695 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "4d439876eac04b058bc1946edf883aa4" format_stamp: "Formatted at 2017-08-19 06:33:37 on dist-test-slave-dist-test-slave-j33z" I0819 06:33:37.982014 1695 fs_manager.cc:280] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s I0819 06:33:37.982162 1695 fs_manager.cc:283] Opened local filesystem: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data,/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/master-0/wal uuid: "4d439876eac04b058bc1946edf883aa4" format_stamp: "Formatted at 2017-08-19 06:33:37 on dist-test-slave-dist-test-slave-j33z" I0819 06:33:37.982364 1695 fs_report.cc:345] Block manager report -------------------- 1 data directories: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0819 06:33:38.151929 1695 master_main.cc:70] Starting Master server... I0819 06:33:38.153149 1712 process_memory.cc:172] Process hard memory limit is 11.777661 GB I0819 06:33:38.153306 1712 process_memory.cc:174] Process soft memory limit is 9.422129 GB I0819 06:33:38.153425 1712 process_memory.cc:177] Process memory pressure threshold is 7.066597 GB I0819 06:33:38.194342 1695 rpc_server.cc:186] RPC server started. Bound to: 127.4.126.254:32833 I0819 06:33:38.234127 1695 webserver.cc:145] Starting webserver on 127.4.126.254:0 I0819 06:33:38.234232 1695 webserver.cc:156] Document root disabled I0819 06:33:38.235517 1695 webserver.cc:278] Webserver started. Bound to: http://127.4.126.254:40062/ I0819 06:33:38.236310 1695 server_base.cc:359] Dumped server information to /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data/info.pb I0819 06:33:38.244444 1150 external_mini_cluster.cc:814] Started /tmp/run_tha_test479zHR/build/tsan/bin/kudu-master as pid 1695 I0819 06:33:38.246098 1150 external_mini_cluster.cc:752] Running /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data --rpc_bind_addresses=127.4.126.1:0 --local_ip_for_outbound_sockets=127.4.126.1 --webserver_interface=127.4.126.1 --webserver_port=0 --tserver_master_addrs=127.4.126.254:32833 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=flag --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=5 --leader_failure_monitor_check_mean_ms=1 --leader_failure_monitor_check_stddev_ms=1 with env {} I0819 06:33:38.256763 1756 tablet_bootstrap.cc:402] T 00000000000000000000000000000000 P 4d439876eac04b058bc1946edf883aa4: Bootstrap starting. I0819 06:33:38.259274 1756 tablet_bootstrap.cc:537] T 00000000000000000000000000000000 P 4d439876eac04b058bc1946edf883aa4: No blocks or log segments found. Creating new log. I0819 06:33:38.260576 1756 log.cc:511] T 00000000000000000000000000000000 P 4d439876eac04b058bc1946edf883aa4: Log is configured to *not* fsync() on all Append() calls I0819 06:33:38.263216 1756 tablet_bootstrap.cc:402] T 00000000000000000000000000000000 P 4d439876eac04b058bc1946edf883aa4: No bootstrap required, opened a new log I0819 06:33:38.269863 1756 raft_consensus.cc:286] T 00000000000000000000000000000000 P 4d439876eac04b058bc1946edf883aa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d439876eac04b058bc1946edf883aa4" member_type: VOTER } I0819 06:33:38.270431 1756 raft_consensus.cc:318] T 00000000000000000000000000000000 P 4d439876eac04b058bc1946edf883aa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0819 06:33:38.270581 1756 raft_consensus.cc:574] T 00000000000000000000000000000000 P 4d439876eac04b058bc1946edf883aa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d439876eac04b058bc1946edf883aa4, State: Running, Role: FOLLOWER I0819 06:33:38.270823 1756 consensus_queue.cc:190] T 00000000000000000000000000000000 P 4d439876eac04b058bc1946edf883aa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0819 06:33:38.271072 1756 raft_consensus.cc:329] T 00000000000000000000000000000000 P 4d439876eac04b058bc1946edf883aa4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0819 06:33:38.271242 1756 raft_consensus.cc:413] T 00000000000000000000000000000000 P 4d439876eac04b058bc1946edf883aa4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0819 06:33:38.271411 1756 raft_consensus.cc:2401] T 00000000000000000000000000000000 P 4d439876eac04b058bc1946edf883aa4: Snoozing failure detection for election timeout plus an additional 1.851s I0819 06:33:38.271549 1756 raft_consensus.cc:2448] T 00000000000000000000000000000000 P 4d439876eac04b058bc1946edf883aa4 [term 0 FOLLOWER]: Advancing to term 1 I0819 06:33:38.274263 1756 raft_consensus.cc:437] T 00000000000000000000000000000000 P 4d439876eac04b058bc1946edf883aa4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d439876eac04b058bc1946edf883aa4" member_type: VOTER } I0819 06:33:38.274755 1756 leader_election.cc:244] T 00000000000000000000000000000000 P 4d439876eac04b058bc1946edf883aa4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0819 06:33:38.276451 1760 raft_consensus.cc:2401] T 00000000000000000000000000000000 P 4d439876eac04b058bc1946edf883aa4: Snoozing failure detection for election timeout plus an additional 1.965s I0819 06:33:38.276724 1760 raft_consensus.cc:2205] T 00000000000000000000000000000000 P 4d439876eac04b058bc1946edf883aa4 [term 1 FOLLOWER]: Leader election won for term 1 I0819 06:33:38.278733 1760 raft_consensus.cc:540] T 00000000000000000000000000000000 P 4d439876eac04b058bc1946edf883aa4 [term 1 LEADER]: Becoming Leader. State: Replica: 4d439876eac04b058bc1946edf883aa4, State: Running, Role: LEADER I0819 06:33:38.279131 1760 consensus_queue.cc:169] T 00000000000000000000000000000000 P 4d439876eac04b058bc1946edf883aa4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d439876eac04b058bc1946edf883aa4" member_type: VOTER } I0819 06:33:38.298665 1756 sys_catalog.cc:287] T 00000000000000000000000000000000 P 4d439876eac04b058bc1946edf883aa4 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "4d439876eac04b058bc1946edf883aa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d439876eac04b058bc1946edf883aa4" member_type: VOTER } } I0819 06:33:38.299084 1756 sys_catalog.cc:290] T 00000000000000000000000000000000 P 4d439876eac04b058bc1946edf883aa4 [sys.catalog]: This master's current role is: LEADER I0819 06:33:38.300125 1762 sys_catalog.cc:287] T 00000000000000000000000000000000 P 4d439876eac04b058bc1946edf883aa4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d439876eac04b058bc1946edf883aa4. Latest consensus state: current_term: 1 leader_uuid: "4d439876eac04b058bc1946edf883aa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d439876eac04b058bc1946edf883aa4" member_type: VOTER } } I0819 06:33:38.300581 1762 sys_catalog.cc:290] T 00000000000000000000000000000000 P 4d439876eac04b058bc1946edf883aa4 [sys.catalog]: This master's current role is: LEADER I0819 06:33:38.300951 1761 sys_catalog.cc:287] T 00000000000000000000000000000000 P 4d439876eac04b058bc1946edf883aa4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d439876eac04b058bc1946edf883aa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d439876eac04b058bc1946edf883aa4" member_type: VOTER } } I0819 06:33:38.301327 1761 sys_catalog.cc:290] T 00000000000000000000000000000000 P 4d439876eac04b058bc1946edf883aa4 [sys.catalog]: This master's current role is: LEADER I0819 06:33:38.303194 1756 sys_catalog.cc:369] T 00000000000000000000000000000000 P 4d439876eac04b058bc1946edf883aa4 [sys.catalog]: configured and running, proceeding with master startup. I0819 06:33:38.303491 1764 catalog_manager.cc:1018] Loading table and tablet metadata into memory... I0819 06:33:38.305069 1695 master_main.cc:73] Master server successfully started. I0819 06:33:38.309445 1764 catalog_manager.cc:1029] Initializing Kudu internal certificate authority... I0819 06:33:38.359534 1764 catalog_manager.cc:914] Generated new certificate authority record I0819 06:33:38.362025 1764 catalog_manager.cc:1038] Loading token signing keys... I0819 06:33:38.371325 1764 catalog_manager.cc:3605] T 00000000000000000000000000000000 P 4d439876eac04b058bc1946edf883aa4: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0819 06:33:38.388057 1757 flags.cc:384] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0819 06:33:38.388483 1757 flags.cc:384] Enabled unsafe flag: --never_fsync=true W0819 06:33:38.403280 1757 flags.cc:384] Enabled experimental flag: --leader_failure_monitor_check_mean_ms=1 W0819 06:33:38.403430 1757 flags.cc:384] Enabled experimental flag: --leader_failure_monitor_check_stddev_ms=1 W0819 06:33:38.403573 1757 flags.cc:384] Enabled experimental flag: --ipki_server_key_size=1024 W0819 06:33:38.403899 1757 flags.cc:384] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.126.1 I0819 06:33:38.424831 1757 tablet_server_main.cc:64] Tablet server non-default flags: --leader_failure_monitor_check_mean_ms=1 --leader_failure_monitor_check_stddev_ms=1 --raft_heartbeat_interval_ms=5 --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.126.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.4.126.1 --webserver_port=0 --tserver_master_addrs=127.4.126.254:32833 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.1757 --redact=flag --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.126.1 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.5.0-SNAPSHOT revision ba0f21e2316b78f167bbaba44cab392558d97b6f build type FASTDEBUG built by None at 19 Aug 2017 06:31:02 UTC on jenkins-slave-c3oe build id 9422 TSAN enabled I0819 06:33:38.435490 1757 tablet_server_main.cc:71] Initializing tablet server... I0819 06:33:38.437350 1757 system_ntp.cc:96] NTP initialized. Resolution in nanos?: 1 Skew PPM: 500 Current error: 1283405 I0819 06:33:38.437970 1757 env_posix.cc:1586] Not raising process file limit of 65536; it is already as high as it can go I0819 06:33:38.438246 1757 file_cache.cc:464] Constructed file cache lbm with capacity 26214 I0819 06:33:38.443370 1757 server_base.cc:215] Could not load existing FS layout: Not found: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0819 06:33:38.443516 1757 server_base.cc:216] Creating new FS layout I0819 06:33:38.451539 1757 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "54716c80d5de4d238c121acf92f30ec1" format_stamp: "Formatted at 2017-08-19 06:33:38 on dist-test-slave-dist-test-slave-j33z" I0819 06:33:38.452289 1757 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "54716c80d5de4d238c121acf92f30ec1" format_stamp: "Formatted at 2017-08-19 06:33:38 on dist-test-slave-dist-test-slave-j33z" I0819 06:33:38.463788 1757 fs_manager.cc:280] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s I0819 06:33:38.463937 1757 fs_manager.cc:283] Opened local filesystem: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data,/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/wal uuid: "54716c80d5de4d238c121acf92f30ec1" format_stamp: "Formatted at 2017-08-19 06:33:38 on dist-test-slave-dist-test-slave-j33z" I0819 06:33:38.464135 1757 fs_report.cc:345] Block manager report -------------------- 1 data directories: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0819 06:33:38.704340 1757 ts_tablet_manager.cc:190] Loaded tablet metadata (0 live tablets) I0819 06:33:38.705869 1757 tablet_server_main.cc:76] Starting tablet server... I0819 06:33:38.816838 1757 rpc_server.cc:186] RPC server started. Bound to: 127.4.126.1:33172 I0819 06:33:38.817742 1757 webserver.cc:145] Starting webserver on 127.4.126.1:0 I0819 06:33:38.817847 1757 webserver.cc:156] Document root disabled I0819 06:33:38.819313 1757 webserver.cc:278] Webserver started. Bound to: http://127.4.126.1:55822/ I0819 06:33:38.820060 1757 server_base.cc:359] Dumped server information to /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/info.pb I0819 06:33:38.824761 1150 external_mini_cluster.cc:814] Started /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver as pid 1757 I0819 06:33:38.825965 1150 external_mini_cluster.cc:752] Running /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data --rpc_bind_addresses=127.4.126.2:0 --local_ip_for_outbound_sockets=127.4.126.2 --webserver_interface=127.4.126.2 --webserver_port=0 --tserver_master_addrs=127.4.126.254:32833 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=flag --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=5 --leader_failure_monitor_check_mean_ms=1 --leader_failure_monitor_check_stddev_ms=1 with env {} I0819 06:33:38.830917 1757 tablet_server_main.cc:79] Tablet server successfully started. I0819 06:33:38.863692 1889 heartbeater.cc:295] Connected to a master server at 127.4.126.254:32833 I0819 06:33:38.864009 1889 heartbeater.cc:365] Registering TS with master... I0819 06:33:38.864619 1889 heartbeater.cc:392] Master 127.4.126.254:32833 requested a full tablet report, sending... I0819 06:33:38.866734 1723 ts_manager.cc:79] Registered new tserver with Master: 54716c80d5de4d238c121acf92f30ec1 (127.4.126.1:33172) I0819 06:33:38.869151 1723 master_service.cc:197] Signed X509 certificate for tserver {username='slave'} at 127.4.126.1:46477 I0819 06:33:38.831768 1890 process_memory.cc:172] Process hard memory limit is 11.777661 GB I0819 06:33:38.872335 1890 process_memory.cc:174] Process soft memory limit is 9.422129 GB I0819 06:33:38.872550 1890 process_memory.cc:177] Process memory pressure threshold is 7.066597 GB WARNING: Logging before InitGoogleLogging() is written to STDERR W0819 06:33:38.965217 1893 flags.cc:384] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0819 06:33:38.965659 1893 flags.cc:384] Enabled unsafe flag: --never_fsync=true W0819 06:33:38.983461 1893 flags.cc:384] Enabled experimental flag: --leader_failure_monitor_check_mean_ms=1 W0819 06:33:38.983636 1893 flags.cc:384] Enabled experimental flag: --leader_failure_monitor_check_stddev_ms=1 W0819 06:33:38.983798 1893 flags.cc:384] Enabled experimental flag: --ipki_server_key_size=1024 W0819 06:33:38.984102 1893 flags.cc:384] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.126.2 I0819 06:33:39.004864 1893 tablet_server_main.cc:64] Tablet server non-default flags: --leader_failure_monitor_check_mean_ms=1 --leader_failure_monitor_check_stddev_ms=1 --raft_heartbeat_interval_ms=5 --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.126.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.4.126.2 --webserver_port=0 --tserver_master_addrs=127.4.126.254:32833 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.1893 --redact=flag --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.126.2 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.5.0-SNAPSHOT revision ba0f21e2316b78f167bbaba44cab392558d97b6f build type FASTDEBUG built by None at 19 Aug 2017 06:31:02 UTC on jenkins-slave-c3oe build id 9422 TSAN enabled I0819 06:33:39.015565 1893 tablet_server_main.cc:71] Initializing tablet server... I0819 06:33:39.017441 1893 system_ntp.cc:96] NTP initialized. Resolution in nanos?: 1 Skew PPM: 500 Current error: 1283905 I0819 06:33:39.018098 1893 env_posix.cc:1586] Not raising process file limit of 65536; it is already as high as it can go I0819 06:33:39.018421 1893 file_cache.cc:464] Constructed file cache lbm with capacity 26214 I0819 06:33:39.023581 1893 server_base.cc:215] Could not load existing FS layout: Not found: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0819 06:33:39.023733 1893 server_base.cc:216] Creating new FS layout I0819 06:33:39.031294 1893 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "6d6a239df0a0404cb3455e68efcb792b" format_stamp: "Formatted at 2017-08-19 06:33:39 on dist-test-slave-dist-test-slave-j33z" I0819 06:33:39.032083 1893 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "6d6a239df0a0404cb3455e68efcb792b" format_stamp: "Formatted at 2017-08-19 06:33:39 on dist-test-slave-dist-test-slave-j33z" I0819 06:33:39.043619 1893 fs_manager.cc:280] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s I0819 06:33:39.043776 1893 fs_manager.cc:283] Opened local filesystem: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data,/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/wal uuid: "6d6a239df0a0404cb3455e68efcb792b" format_stamp: "Formatted at 2017-08-19 06:33:39 on dist-test-slave-dist-test-slave-j33z" I0819 06:33:39.043975 1893 fs_report.cc:345] Block manager report -------------------- 1 data directories: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0819 06:33:39.292639 1893 ts_tablet_manager.cc:190] Loaded tablet metadata (0 live tablets) I0819 06:33:39.294134 1893 tablet_server_main.cc:76] Starting tablet server... I0819 06:33:39.404497 1893 rpc_server.cc:186] RPC server started. Bound to: 127.4.126.2:53826 I0819 06:33:39.405398 1893 webserver.cc:145] Starting webserver on 127.4.126.2:0 I0819 06:33:39.405505 1893 webserver.cc:156] Document root disabled I0819 06:33:39.406914 1893 webserver.cc:278] Webserver started. Bound to: http://127.4.126.2:40930/ I0819 06:33:39.407673 1893 server_base.cc:359] Dumped server information to /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/info.pb I0819 06:33:39.416184 1150 external_mini_cluster.cc:814] Started /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver as pid 1893 I0819 06:33:39.415961 2015 process_memory.cc:172] Process hard memory limit is 11.777661 GB I0819 06:33:39.416494 2015 process_memory.cc:174] Process soft memory limit is 9.422129 GB I0819 06:33:39.417031 2015 process_memory.cc:177] Process memory pressure threshold is 7.066597 GB I0819 06:33:39.417469 1150 external_mini_cluster.cc:752] Running /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data --rpc_bind_addresses=127.4.126.3:0 --local_ip_for_outbound_sockets=127.4.126.3 --webserver_interface=127.4.126.3 --webserver_port=0 --tserver_master_addrs=127.4.126.254:32833 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=flag --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=5 --leader_failure_monitor_check_mean_ms=1 --leader_failure_monitor_check_stddev_ms=1 with env {} I0819 06:33:39.440958 2014 heartbeater.cc:295] Connected to a master server at 127.4.126.254:32833 I0819 06:33:39.441292 2014 heartbeater.cc:365] Registering TS with master... I0819 06:33:39.441864 2014 heartbeater.cc:392] Master 127.4.126.254:32833 requested a full tablet report, sending... I0819 06:33:39.443778 1723 ts_manager.cc:79] Registered new tserver with Master: 6d6a239df0a0404cb3455e68efcb792b (127.4.126.2:53826) I0819 06:33:39.445365 1723 master_service.cc:197] Signed X509 certificate for tserver {username='slave'} at 127.4.126.2:34672 I0819 06:33:39.416256 1893 tablet_server_main.cc:79] Tablet server successfully started. WARNING: Logging before InitGoogleLogging() is written to STDERR W0819 06:33:39.552606 2018 flags.cc:384] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0819 06:33:39.553004 2018 flags.cc:384] Enabled unsafe flag: --never_fsync=true W0819 06:33:39.568061 2018 flags.cc:384] Enabled experimental flag: --leader_failure_monitor_check_mean_ms=1 W0819 06:33:39.568220 2018 flags.cc:384] Enabled experimental flag: --leader_failure_monitor_check_stddev_ms=1 W0819 06:33:39.568361 2018 flags.cc:384] Enabled experimental flag: --ipki_server_key_size=1024 W0819 06:33:39.568681 2018 flags.cc:384] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.126.3 I0819 06:33:39.588980 2018 tablet_server_main.cc:64] Tablet server non-default flags: --leader_failure_monitor_check_mean_ms=1 --leader_failure_monitor_check_stddev_ms=1 --raft_heartbeat_interval_ms=5 --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.126.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.4.126.3 --webserver_port=0 --tserver_master_addrs=127.4.126.254:32833 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.2018 --redact=flag --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.126.3 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.5.0-SNAPSHOT revision ba0f21e2316b78f167bbaba44cab392558d97b6f build type FASTDEBUG built by None at 19 Aug 2017 06:31:02 UTC on jenkins-slave-c3oe build id 9422 TSAN enabled I0819 06:33:39.599553 2018 tablet_server_main.cc:71] Initializing tablet server... I0819 06:33:39.601388 2018 system_ntp.cc:96] NTP initialized. Resolution in nanos?: 1 Skew PPM: 500 Current error: 1283905 I0819 06:33:39.602072 2018 env_posix.cc:1586] Not raising process file limit of 65536; it is already as high as it can go I0819 06:33:39.602347 2018 file_cache.cc:464] Constructed file cache lbm with capacity 26214 I0819 06:33:39.607395 2018 server_base.cc:215] Could not load existing FS layout: Not found: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0819 06:33:39.607549 2018 server_base.cc:216] Creating new FS layout I0819 06:33:39.615154 2018 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "6f8e517e75d440519fb9d468724d7798" format_stamp: "Formatted at 2017-08-19 06:33:39 on dist-test-slave-dist-test-slave-j33z" I0819 06:33:39.615939 2018 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "6f8e517e75d440519fb9d468724d7798" format_stamp: "Formatted at 2017-08-19 06:33:39 on dist-test-slave-dist-test-slave-j33z" I0819 06:33:39.627413 2018 fs_manager.cc:280] Time spent opening block manager: real 0.004s user 0.000s sys 0.008s I0819 06:33:39.627576 2018 fs_manager.cc:283] Opened local filesystem: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data,/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/wal uuid: "6f8e517e75d440519fb9d468724d7798" format_stamp: "Formatted at 2017-08-19 06:33:39 on dist-test-slave-dist-test-slave-j33z" I0819 06:33:39.627779 2018 fs_report.cc:345] Block manager report -------------------- 1 data directories: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0819 06:33:39.817190 2018 ts_tablet_manager.cc:190] Loaded tablet metadata (0 live tablets) I0819 06:33:39.818346 2018 tablet_server_main.cc:76] Starting tablet server... I0819 06:33:39.873183 1889 heartbeater.cc:384] Master 127.4.126.254:32833 was elected leader, sending a full tablet report... I0819 06:33:39.912345 2018 rpc_server.cc:186] RPC server started. Bound to: 127.4.126.3:53370 I0819 06:33:39.913292 2018 webserver.cc:145] Starting webserver on 127.4.126.3:0 I0819 06:33:39.913406 2018 webserver.cc:156] Document root disabled I0819 06:33:39.914674 2018 webserver.cc:278] Webserver started. Bound to: http://127.4.126.3:48504/ I0819 06:33:39.915473 2018 server_base.cc:359] Dumped server information to /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/info.pb I0819 06:33:39.917414 1150 external_mini_cluster.cc:814] Started /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver as pid 2018 I0819 06:33:39.918208 2140 process_memory.cc:172] Process hard memory limit is 11.777661 GB I0819 06:33:39.918380 2140 process_memory.cc:174] Process soft memory limit is 9.422129 GB I0819 06:33:39.918500 2140 process_memory.cc:177] Process memory pressure threshold is 7.066597 GB I0819 06:33:39.921123 2018 tablet_server_main.cc:79] Tablet server successfully started. I0819 06:33:39.940032 2139 heartbeater.cc:295] Connected to a master server at 127.4.126.254:32833 I0819 06:33:39.940300 2139 heartbeater.cc:365] Registering TS with master... I0819 06:33:39.940799 2139 heartbeater.cc:392] Master 127.4.126.254:32833 requested a full tablet report, sending... I0819 06:33:39.942298 1723 ts_manager.cc:79] Registered new tserver with Master: 6f8e517e75d440519fb9d468724d7798 (127.4.126.3:53370) I0819 06:33:39.943450 1723 master_service.cc:197] Signed X509 certificate for tserver {username='slave'} at 127.4.126.3:38994 I0819 06:33:39.945097 1150 external_mini_cluster.cc:454] 3 TS(s) registered with all masters I0819 06:33:39.947597 1150 test_util.cc:185] Using random seed: 1830464564 I0819 06:33:39.965322 1723 catalog_manager.cc:1281] CreateTable from {username='slave'} at 127.0.0.1:41910: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0819 06:33:40.012456 1951 tablet_service.cc:684] Processing CreateTablet for tablet 8b79633d21dc49868650b1a34ca1c8ea (table=test-workload [id=85ad8f64c0634f04a3ee6dc1b577b941]), partition=RANGE (key) PARTITION UNBOUNDED I0819 06:33:40.013151 2076 tablet_service.cc:684] Processing CreateTablet for tablet 8b79633d21dc49868650b1a34ca1c8ea (table=test-workload [id=85ad8f64c0634f04a3ee6dc1b577b941]), partition=RANGE (key) PARTITION UNBOUNDED I0819 06:33:40.013947 1826 tablet_service.cc:684] Processing CreateTablet for tablet 8b79633d21dc49868650b1a34ca1c8ea (table=test-workload [id=85ad8f64c0634f04a3ee6dc1b577b941]), partition=RANGE (key) PARTITION UNBOUNDED I0819 06:33:40.025023 1826 ts_tablet_manager.cc:877] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Registered tablet (data state: TABLET_DATA_READY) I0819 06:33:40.026692 2159 ts_tablet_manager.cc:740] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Bootstrapping tablet I0819 06:33:40.027215 2159 tablet_bootstrap.cc:402] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Bootstrap starting. I0819 06:33:40.028934 1951 ts_tablet_manager.cc:877] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b: Registered tablet (data state: TABLET_DATA_READY) I0819 06:33:40.029938 2159 tablet_bootstrap.cc:537] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: No blocks or log segments found. Creating new log. I0819 06:33:40.030833 2160 ts_tablet_manager.cc:740] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b: Bootstrapping tablet I0819 06:33:40.031361 2160 tablet_bootstrap.cc:402] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b: Bootstrap starting. I0819 06:33:40.031561 2159 log.cc:511] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Log is configured to *not* fsync() on all Append() calls I0819 06:33:40.034524 2160 tablet_bootstrap.cc:537] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b: No blocks or log segments found. Creating new log. I0819 06:33:40.034540 2076 ts_tablet_manager.cc:877] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Registered tablet (data state: TABLET_DATA_READY) I0819 06:33:40.034940 2159 tablet_bootstrap.cc:402] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: No bootstrap required, opened a new log I0819 06:33:40.035382 2159 ts_tablet_manager.cc:752] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s I0819 06:33:40.036283 2160 log.cc:511] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b: Log is configured to *not* fsync() on all Append() calls I0819 06:33:40.038774 2162 ts_tablet_manager.cc:740] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Bootstrapping tablet I0819 06:33:40.038650 2159 raft_consensus.cc:286] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:40.039280 2162 tablet_bootstrap.cc:402] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Bootstrap starting. I0819 06:33:40.039881 2160 tablet_bootstrap.cc:402] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b: No bootstrap required, opened a new log I0819 06:33:40.039876 2159 raft_consensus.cc:318] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0819 06:33:40.040112 2159 raft_consensus.cc:574] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54716c80d5de4d238c121acf92f30ec1, State: Running, Role: FOLLOWER I0819 06:33:40.040304 2160 ts_tablet_manager.cc:752] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s I0819 06:33:40.040482 2159 consensus_queue.cc:190] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0819 06:33:40.042049 2162 tablet_bootstrap.cc:537] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: No blocks or log segments found. Creating new log. I0819 06:33:40.043402 2162 log.cc:511] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Log is configured to *not* fsync() on all Append() calls I0819 06:33:40.044687 2160 raft_consensus.cc:286] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:40.045756 2160 raft_consensus.cc:318] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0819 06:33:40.045941 2160 raft_consensus.cc:574] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d6a239df0a0404cb3455e68efcb792b, State: Running, Role: FOLLOWER I0819 06:33:40.046221 2160 consensus_queue.cc:190] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0819 06:33:40.048369 2014 heartbeater.cc:384] Master 127.4.126.254:32833 was elected leader, sending a full tablet report... I0819 06:33:40.048964 2164 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0819 06:33:40.049226 2164 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.015s I0819 06:33:40.049384 2164 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:40.050160 2160 ts_tablet_manager.cc:781] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s I0819 06:33:40.050781 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:40.051403 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 6f8e517e75d440519fb9d468724d7798 I0819 06:33:40.055284 2166 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0819 06:33:40.055685 2166 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b: Snoozing failure detection for election timeout plus an additional 0.019s I0819 06:33:40.055934 2166 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:40.057472 2162 tablet_bootstrap.cc:402] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: No bootstrap required, opened a new log I0819 06:33:40.057920 2162 ts_tablet_manager.cc:752] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.012s I0819 06:33:40.061362 2166 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 54716c80d5de4d238c121acf92f30ec1 I0819 06:33:40.061982 2166 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 6f8e517e75d440519fb9d468724d7798 W0819 06:33:40.068055 1779 leader_election.cc:273] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 6f8e517e75d440519fb9d468724d7798: Timed out: connection negotiation to 127.4.126.3:53370 for RPC RequestConsensusVote timed out after 0.015s (ON_OUTBOUND_QUEUE) I0819 06:33:40.067461 2162 raft_consensus.cc:286] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:40.069406 2162 raft_consensus.cc:318] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0819 06:33:40.069660 2162 raft_consensus.cc:574] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f8e517e75d440519fb9d468724d7798, State: Running, Role: FOLLOWER I0819 06:33:40.070036 2162 consensus_queue.cc:190] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER W0819 06:33:40.072240 1781 leader_election.cc:273] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 6d6a239df0a0404cb3455e68efcb792b: Timed out: connection negotiation to 127.4.126.2:53826 for RPC RequestConsensusVote timed out after 0.015s (ON_OUTBOUND_QUEUE) I0819 06:33:40.072563 1781 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0819 06:33:40.074283 2159 ts_tablet_manager.cc:781] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Time spent starting tablet: real 0.039s user 0.012s sys 0.012s I0819 06:33:40.079532 2171 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0819 06:33:40.079772 2171 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.021s I0819 06:33:40.079946 2171 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:40.081035 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:40.081483 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 54716c80d5de4d238c121acf92f30ec1 W0819 06:33:40.083209 1903 leader_election.cc:273] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 54716c80d5de4d238c121acf92f30ec1: Timed out: connection negotiation to 127.4.126.1:33172 for RPC RequestConsensusVote timed out after 0.019s (ON_OUTBOUND_QUEUE) W0819 06:33:40.083818 1904 leader_election.cc:273] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 6f8e517e75d440519fb9d468724d7798: Timed out: connection negotiation to 127.4.126.3:53370 for RPC RequestConsensusVote timed out after 0.019s (ON_OUTBOUND_QUEUE) I0819 06:33:40.084050 1904 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0819 06:33:40.084640 2168 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b: Snoozing failure detection for election timeout plus an additional 0.021s I0819 06:33:40.084895 2168 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0819 06:33:40.088994 2164 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0819 06:33:40.089175 2139 heartbeater.cc:384] Master 127.4.126.254:32833 was elected leader, sending a full tablet report... I0819 06:33:40.089279 2164 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.016s I0819 06:33:40.089496 2164 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:40.090486 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:40.090886 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 6f8e517e75d440519fb9d468724d7798 I0819 06:33:40.092774 2096 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.017s I0819 06:33:40.093008 2096 raft_consensus.cc:1967] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54716c80d5de4d238c121acf92f30ec1 in term 0. I0819 06:33:40.094457 1971 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b: Snoozing failure detection for election timeout plus an additional 0.027s I0819 06:33:40.094707 1971 raft_consensus.cc:1967] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f8e517e75d440519fb9d468724d7798 in term 0. I0819 06:33:40.095824 1779 leader_election.cc:349] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 6f8e517e75d440519fb9d468724d7798 I0819 06:33:40.096016 1779 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0819 06:33:40.096624 2031 leader_election.cc:349] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:40.096845 2031 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0819 06:33:40.098731 2178 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.016s I0819 06:33:40.099112 2178 raft_consensus.cc:2205] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0819 06:33:40.099360 2178 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0819 06:33:40.099596 2178 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.018s I0819 06:33:40.099822 2178 raft_consensus.cc:2448] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 0 FOLLOWER]: Advancing to term 1 I0819 06:33:40.103235 2178 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:40.104251 2178 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 1 election: Requesting vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:40.105700 2178 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 1 election: Requesting vote from peer 54716c80d5de4d238c121acf92f30ec1 I0819 06:33:40.105942 1971 raft_consensus.cc:2448] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 0 FOLLOWER]: Advancing to term 1 I0819 06:33:40.106343 1971 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b: Snoozing failure detection for election timeout plus an additional 0.033s W0819 06:33:40.106495 2028 leader_election.cc:273] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 54716c80d5de4d238c121acf92f30ec1: Timed out: connection negotiation to 127.4.126.1:33172 for RPC RequestConsensusVote timed out after 0.021s (ON_OUTBOUND_QUEUE) I0819 06:33:40.109267 1971 raft_consensus.cc:1967] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f8e517e75d440519fb9d468724d7798 in term 1. I0819 06:33:40.121840 2164 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0819 06:33:40.122066 2164 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.016s I0819 06:33:40.122246 2164 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:40.122987 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:40.123298 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 6f8e517e75d440519fb9d468724d7798 W0819 06:33:40.129793 1781 outbound_call.cc:308] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.126.2:53826, user_credentials={real_user=slave}} blocked reactor thread for 57760.7us I0819 06:33:40.132019 1971 raft_consensus.cc:1897] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 in current term 1: Already voted for candidate 6f8e517e75d440519fb9d468724d7798 in this term. I0819 06:33:40.133919 1971 raft_consensus.cc:1897] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 in current term 1: Already voted for candidate 6f8e517e75d440519fb9d468724d7798 in this term. I0819 06:33:40.137689 1781 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 in current term 1: Already voted for candidate 6f8e517e75d440519fb9d468724d7798 in this term. I0819 06:33:40.138849 1781 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 in current term 1: Already voted for candidate 6f8e517e75d440519fb9d468724d7798 in this term. I0819 06:33:40.142688 2176 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.019s W0819 06:33:40.142776 1779 outbound_call.cc:308] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.126.3:53370, user_credentials={real_user=slave}} blocked reactor thread for 46975.3us I0819 06:33:40.142973 2176 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0819 06:33:40.143237 2176 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.020s I0819 06:33:40.143426 2176 raft_consensus.cc:2205] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0819 06:33:40.143575 2176 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0819 06:33:40.143697 2176 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.029s I0819 06:33:40.143803 2176 raft_consensus.cc:2448] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 0 FOLLOWER]: Advancing to term 1 I0819 06:33:40.146841 2176 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:40.147678 2176 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 1 election: Requesting vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:40.148010 2176 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 1 election: Requesting vote from peer 6f8e517e75d440519fb9d468724d7798 W0819 06:33:40.160019 1779 leader_election.cc:273] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 6f8e517e75d440519fb9d468724d7798: Timed out: RequestConsensusVote RPC to 127.4.126.3:53370 timed out after 0.016s (SENT) I0819 06:33:40.160214 1779 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0819 06:33:40.160712 2165 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.033s I0819 06:33:40.160928 2165 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0819 06:33:40.161415 1971 raft_consensus.cc:1897] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 in current term 1: Already voted for candidate 6f8e517e75d440519fb9d468724d7798 in this term. I0819 06:33:40.162286 1781 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 1 election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 in current term 1: Already voted for candidate 6f8e517e75d440519fb9d468724d7798 in this term. I0819 06:33:40.166607 2166 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0819 06:33:40.166898 2166 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b: Snoozing failure detection for election timeout plus an additional 0.019s I0819 06:33:40.167075 2166 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:40.167907 2166 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 54716c80d5de4d238c121acf92f30ec1 I0819 06:33:40.168282 2166 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 6f8e517e75d440519fb9d468724d7798 I0819 06:33:40.169832 1846 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.049s I0819 06:33:40.170085 1846 raft_consensus.cc:1967] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d6a239df0a0404cb3455e68efcb792b in term 1. I0819 06:33:40.171073 1903 leader_election.cc:349] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [CANDIDATE]: Term 2 pre-election: Vote granted by peer 54716c80d5de4d238c121acf92f30ec1 I0819 06:33:40.171228 1903 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0819 06:33:40.171587 2168 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b: Snoozing failure detection for election timeout plus an additional 0.024s I0819 06:33:40.171768 2168 raft_consensus.cc:2205] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 1 FOLLOWER]: Leader pre-election won for term 2 I0819 06:33:40.171907 2168 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0819 06:33:40.172065 2168 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b: Snoozing failure detection for election timeout plus an additional 0.018s I0819 06:33:40.172199 2168 raft_consensus.cc:2448] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 1 FOLLOWER]: Advancing to term 2 I0819 06:33:40.175213 2168 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:40.176015 2168 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [CANDIDATE]: Term 2 election: Requesting vote from peer 54716c80d5de4d238c121acf92f30ec1 I0819 06:33:40.176367 2168 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [CANDIDATE]: Term 2 election: Requesting vote from peer 6f8e517e75d440519fb9d468724d7798 W0819 06:33:40.177386 1779 leader_election.cc:273] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 6f8e517e75d440519fb9d468724d7798: Timed out: RequestConsensusVote RPC to 127.4.126.3:53370 timed out after 0.029s (SENT) I0819 06:33:40.177594 1846 raft_consensus.cc:2448] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 1 FOLLOWER]: Advancing to term 2 I0819 06:33:40.177666 1779 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I0819 06:33:40.177958 1846 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.023s I0819 06:33:40.182098 1846 raft_consensus.cc:1967] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d6a239df0a0404cb3455e68efcb792b in term 2. I0819 06:33:40.182494 2179 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.027s I0819 06:33:40.182749 2179 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority I0819 06:33:40.183269 1903 leader_election.cc:349] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [CANDIDATE]: Term 2 election: Vote granted by peer 54716c80d5de4d238c121acf92f30ec1 I0819 06:33:40.183503 1903 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0819 06:33:40.183871 2168 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b: Snoozing failure detection for election timeout plus an additional 0.016s I0819 06:33:40.184062 2168 raft_consensus.cc:2205] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 FOLLOWER]: Leader election won for term 2 W0819 06:33:40.188594 1904 leader_election.cc:273] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f8e517e75d440519fb9d468724d7798: Timed out: RequestConsensusVote RPC to 127.4.126.3:53370 timed out after 0.019s (SENT) W0819 06:33:40.195448 1904 leader_election.cc:273] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6f8e517e75d440519fb9d468724d7798: Timed out: RequestConsensusVote RPC to 127.4.126.3:53370 timed out after 0.018s (SENT) I0819 06:33:40.199770 2168 raft_consensus.cc:540] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Becoming Leader. State: Replica: 6d6a239df0a0404cb3455e68efcb792b, State: Running, Role: LEADER I0819 06:33:40.200235 2168 consensus_queue.cc:169] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:40.210705 1723 catalog_manager.cc:2559] T 8b79633d21dc49868650b1a34ca1c8ea reported consensus state change: term changed from 0 to 2, leader changed from to 6d6a239df0a0404cb3455e68efcb792b (127.4.126.2). New consensus state: current_term: 2 leader_uuid: "6d6a239df0a0404cb3455e68efcb792b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } } I0819 06:33:40.098855 2162 ts_tablet_manager.cc:781] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Time spent starting tablet: real 0.041s user 0.000s sys 0.008s I0819 06:33:40.319753 2164 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:33:40.320034 2164 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.018s I0819 06:33:40.320238 2164 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:40.321506 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:40.321909 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6f8e517e75d440519fb9d468724d7798 I0819 06:33:40.323914 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:40.325171 1781 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:40.110074 2031 leader_election.cc:349] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 1 election: Vote granted by peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:40.328786 2031 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. W0819 06:33:40.329226 2031 outbound_call.cc:308] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.126.2:53826, user_credentials={real_user=slave}} blocked reactor thread for 219158us W0819 06:33:40.330862 2028 leader_election.cc:273] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 54716c80d5de4d238c121acf92f30ec1: Timed out: connection negotiation to 127.4.126.1:33172 for RPC RequestConsensusVote timed out after 0.018s (ON_OUTBOUND_QUEUE) W0819 06:33:40.331527 2028 outbound_call.cc:308] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.126.1:33172, user_credentials={real_user=slave}} blocked reactor thread for 206332us W0819 06:33:40.342053 1779 leader_election.cc:273] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6f8e517e75d440519fb9d468724d7798: Timed out: RequestConsensusVote RPC to 127.4.126.3:53370 timed out after 0.018s (SENT) I0819 06:33:40.342294 1779 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0819 06:33:40.342902 2179 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.020s I0819 06:33:40.343091 2179 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0819 06:33:40.135243 2171 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0819 06:33:40.412173 2171 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.019s I0819 06:33:40.412344 2171 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:40.413157 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:40.413408 2093 raft_consensus.cc:1942] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d6a239df0a0404cb3455e68efcb792b for term 2 because replica is already servicing an update from a current leader or another vote. I0819 06:33:40.413564 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 54716c80d5de4d238c121acf92f30ec1 W0819 06:33:40.414186 2093 rpcz_store.cc:235] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.4.126.2:43447 (request call id 2) took 82ms (client timeout 19). I0819 06:33:40.415316 1846 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 2 because replica is either leader or believes a valid leader to be alive. W0819 06:33:40.415597 2093 rpcz_store.cc:239] Trace: 0819 06:33:40.332222 (+ 0us) service_pool.cc:149] Inserting onto call queue 0819 06:33:40.333973 (+ 1751us) service_pool.cc:208] Handling call 0819 06:33:40.413962 (+ 79989us) spinlock_profiling.cc:255] Waited 79 ms on lock 0x7b480004028c. stack: 00007fa785f5e247 00007fa785f5e140 00007fa784a4a044 00007fa78d8c6e54 00007fa78d8c6de9 00007fa78c4bbd1e 00007fa78d8fd1dd 00007fa78833cce4 00007fa78833cc71 00007fa78833cb84 00007fa7878951a9 00007fa787894bd4 00007fa78789618d 00007fa787897fa6 00007fa787897efa 00007fa787897e83 0819 06:33:40.414140 (+ 178us) inbound_call.cc:149] Queueing success response Metrics: {"spinlock_wait_cycles":181715840} I0819 06:33:40.416543 2028 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 54716c80d5de4d238c121acf92f30ec1. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 2 because replica is either leader or believes a valid leader to be alive. I0819 06:33:40.416859 2096 raft_consensus.cc:1897] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 in current term 1: Already voted for candidate 6f8e517e75d440519fb9d468724d7798 in this term. I0819 06:33:40.417307 2095 raft_consensus.cc:1942] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 1 because replica is already servicing an update from a current leader or another vote. W0819 06:33:40.417345 2096 rpcz_store.cc:235] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.4.126.1:57286 (request call id 3) took 87ms (client timeout 16). W0819 06:33:40.417737 2095 rpcz_store.cc:235] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.4.126.1:57286 (request call id 4) took 87ms (client timeout 29). I0819 06:33:40.418033 2175 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.018s I0819 06:33:40.419210 2092 raft_consensus.cc:1942] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d6a239df0a0404cb3455e68efcb792b for term 2 because replica is already servicing an update from a current leader or another vote. W0819 06:33:40.419330 2096 rpcz_store.cc:239] Trace: 0819 06:33:40.329920 (+ 0us) service_pool.cc:149] Inserting onto call queue 0819 06:33:40.333144 (+ 3224us) service_pool.cc:208] Handling call 0819 06:33:40.417179 (+ 84035us) spinlock_profiling.cc:255] Waited 81.4 ms on lock 0x7b480004028c. stack: 00007fa785f5e247 00007fa785f5e140 00007fa784a4a044 00007fa78d8c6e54 00007fa78d8c6de9 00007fa78c4bbe55 00007fa78d8fd1dd 00007fa78833cce4 00007fa78833cc71 00007fa78833cb84 00007fa7878951a9 00007fa787894bd4 00007fa78789618d 00007fa787897fa6 00007fa787897efa 00007fa787897e83 0819 06:33:40.417292 (+ 113us) inbound_call.cc:149] Queueing success response Metrics: {"spinlock_wait_cycles":187219712} W0819 06:33:40.419698 2092 rpcz_store.cc:235] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.4.126.2:43447 (request call id 3) took 87ms (client timeout 18). I0819 06:33:40.420827 2094 raft_consensus.cc:1942] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is already servicing an update from a current leader or another vote. W0819 06:33:40.420892 2092 rpcz_store.cc:239] Trace: 0819 06:33:40.332574 (+ 0us) service_pool.cc:149] Inserting onto call queue 0819 06:33:40.334255 (+ 1681us) service_pool.cc:208] Handling call 0819 06:33:40.419480 (+ 85225us) spinlock_profiling.cc:255] Waited 84 ms on lock 0x7b480004028c. stack: 00007fa785f5e247 00007fa785f5e140 00007fa784a4a044 00007fa78d8c6e54 00007fa78d8c6de9 00007fa78c4bbd1e 00007fa78d8fd1dd 00007fa78833cce4 00007fa78833cc71 00007fa78833cb84 00007fa7878951a9 00007fa787894bd4 00007fa78789618d 00007fa787897fa6 00007fa787897efa 00007fa787897e83 0819 06:33:40.419655 (+ 175us) inbound_call.cc:149] Queueing success response Metrics: {"spinlock_wait_cycles":193134848} W0819 06:33:40.421378 2095 rpcz_store.cc:239] Trace: 0819 06:33:40.330251 (+ 0us) service_pool.cc:149] Inserting onto call queue 0819 06:33:40.333412 (+ 3161us) service_pool.cc:208] Handling call 0819 06:33:40.417600 (+ 84188us) spinlock_profiling.cc:255] Waited 83.5 ms on lock 0x7b480004028c. stack: 00007fa785f5e247 00007fa785f5e140 00007fa784a4a044 00007fa78d8c6e54 00007fa78d8c6de9 00007fa78c4bbd1e 00007fa78d8fd1dd 00007fa78833cce4 00007fa78833cc71 00007fa78833cb84 00007fa7878951a9 00007fa787894bd4 00007fa78789618d 00007fa787897fa6 00007fa787897efa 00007fa787897e83 0819 06:33:40.417695 (+ 95us) inbound_call.cc:149] Queueing success response Metrics: {"spinlock_wait_cycles":192099200} W0819 06:33:40.421356 2094 rpcz_store.cc:235] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.4.126.1:57286 (request call id 5) took 90ms (client timeout 18). I0819 06:33:40.422066 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 2 because replica is either leader or believes a valid leader to be alive. I0819 06:33:40.422272 2091 raft_consensus.cc:2448] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 1 FOLLOWER]: Advancing to term 2 W0819 06:33:40.423120 2094 rpcz_store.cc:239] Trace: 0819 06:33:40.330560 (+ 0us) service_pool.cc:149] Inserting onto call queue 0819 06:33:40.333666 (+ 3106us) service_pool.cc:208] Handling call 0819 06:33:40.421132 (+ 87466us) spinlock_profiling.cc:255] Waited 85.7 ms on lock 0x7b480004028c. stack: 00007fa785f5e247 00007fa785f5e140 00007fa784a4a044 00007fa78d8c6e54 00007fa78d8c6de9 00007fa78c4bbd1e 00007fa78d8fd1dd 00007fa78833cce4 00007fa78833cc71 00007fa78833cb84 00007fa7878951a9 00007fa787894bd4 00007fa78789618d 00007fa787897fa6 00007fa787897efa 00007fa787897e83 0819 06:33:40.421315 (+ 183us) inbound_call.cc:149] Queueing success response Metrics: {"spinlock_wait_cycles":197184896} I0819 06:33:40.423187 2031 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 2 because replica is either leader or believes a valid leader to be alive. I0819 06:33:40.423434 2031 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0819 06:33:40.437500 2175 raft_consensus.cc:2168] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader election decision vote started in defunct term 1: won I0819 06:33:40.437907 2178 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.019s I0819 06:33:40.438133 2178 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0819 06:33:40.488950 2171 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:33:40.489233 2171 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.027s I0819 06:33:40.489456 2171 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:40.490592 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:40.490972 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 54716c80d5de4d238c121acf92f30ec1 I0819 06:33:40.492640 1845 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:40.493666 2028 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 54716c80d5de4d238c121acf92f30ec1. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:40.497321 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:40.500519 2031 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:40.500768 2031 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0819 06:33:40.501371 2175 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.022s I0819 06:33:40.501595 2175 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0819 06:33:40.832262 2171 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:33:40.832576 2171 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.021s I0819 06:33:40.832792 2171 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:40.833806 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:40.834172 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 54716c80d5de4d238c121acf92f30ec1 I0819 06:33:40.836134 1846 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:40.837324 2028 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 54716c80d5de4d238c121acf92f30ec1. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:40.839226 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:40.839993 2031 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:40.841017 2031 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0819 06:33:40.843365 2178 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.033s I0819 06:33:40.843574 2178 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0819 06:33:42.169623 2171 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:33:42.169891 2171 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.057s I0819 06:33:42.170090 2171 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:42.171174 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:42.171619 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 54716c80d5de4d238c121acf92f30ec1 I0819 06:33:42.175235 1844 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:42.176013 2028 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 54716c80d5de4d238c121acf92f30ec1. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:42.179141 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:42.183667 2031 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:42.183923 2031 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0819 06:33:42.187819 2203 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.036s I0819 06:33:42.187999 2203 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0819 06:33:45.716817 2164 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:33:45.717074 2164 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.030s I0819 06:33:45.717247 2164 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:45.718075 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:45.718394 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6f8e517e75d440519fb9d468724d7798 I0819 06:33:45.719941 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:45.719979 2091 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:45.720932 1779 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6f8e517e75d440519fb9d468724d7798. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:45.721253 1781 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:45.721423 1781 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0819 06:33:45.723604 2204 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.018s I0819 06:33:45.723832 2204 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0819 06:33:45.830976 2164 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:33:45.831249 2164 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.027s I0819 06:33:45.831429 2164 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:45.832250 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:45.832600 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6f8e517e75d440519fb9d468724d7798 I0819 06:33:45.833724 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:45.835496 1781 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:45.835659 2094 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:45.836500 1779 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6f8e517e75d440519fb9d468724d7798. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:45.836724 1779 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0819 06:33:45.837136 2204 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.033s I0819 06:33:45.837326 2204 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0819 06:33:46.719806 2164 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:33:46.720080 2164 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.023s I0819 06:33:46.720293 2164 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:46.721302 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:46.721679 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6f8e517e75d440519fb9d468724d7798 I0819 06:33:46.723516 1971 raft_consensus.cc:1942] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is already servicing an update from a current leader or another vote. I0819 06:33:46.723732 2091 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:46.724524 1781 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Service unavailable: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is already servicing an update from a current leader or another vote. I0819 06:33:46.724870 1779 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6f8e517e75d440519fb9d468724d7798. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:46.725116 1779 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0819 06:33:46.727324 2205 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.053s I0819 06:33:46.727546 2205 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0819 06:33:47.299428 2171 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:33:47.299666 2171 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.092s I0819 06:33:47.299840 2171 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:47.300639 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:47.300972 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 54716c80d5de4d238c121acf92f30ec1 I0819 06:33:47.302767 1846 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:47.304085 2028 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 54716c80d5de4d238c121acf92f30ec1. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:47.304613 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:47.306139 2031 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:47.306380 2031 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0819 06:33:47.309584 2164 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:33:47.309801 2164 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.016s I0819 06:33:47.309988 2164 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:47.310710 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:47.311025 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6f8e517e75d440519fb9d468724d7798 I0819 06:33:47.312207 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:47.313067 2206 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.105s I0819 06:33:47.313311 2206 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0819 06:33:47.313486 1781 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:47.313927 2094 raft_consensus.cc:1942] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is already servicing an update from a current leader or another vote. I0819 06:33:47.314787 1779 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6f8e517e75d440519fb9d468724d7798. Message: Service unavailable: T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is already servicing an update from a current leader or another vote. I0819 06:33:47.315013 1779 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0819 06:33:47.337927 2207 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.048s I0819 06:33:47.338227 2207 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0819 06:33:48.311657 2171 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:33:48.311940 2171 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.076s I0819 06:33:48.312163 2171 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:48.313164 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:48.313580 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 54716c80d5de4d238c121acf92f30ec1 I0819 06:33:48.315557 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:48.318066 2031 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:48.319774 1846 raft_consensus.cc:1942] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is already servicing an update from a current leader or another vote. I0819 06:33:48.321038 2028 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 54716c80d5de4d238c121acf92f30ec1. Message: Service unavailable: T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is already servicing an update from a current leader or another vote. I0819 06:33:48.321321 2028 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0819 06:33:48.335219 2171 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:33:48.335513 2171 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.070s I0819 06:33:48.335718 2171 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:48.339156 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:48.340029 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 54716c80d5de4d238c121acf92f30ec1 I0819 06:33:48.341919 2208 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.082s I0819 06:33:48.342190 2208 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0819 06:33:48.345047 1844 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:48.347086 2028 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 54716c80d5de4d238c121acf92f30ec1. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:48.347995 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:48.350625 2031 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:48.350859 2031 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0819 06:33:48.352643 2208 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.238s I0819 06:33:48.352885 2208 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0819 06:33:49.301800 2164 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:33:49.302085 2164 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.044s I0819 06:33:49.302317 2164 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:49.303323 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:49.303722 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6f8e517e75d440519fb9d468724d7798 I0819 06:33:49.306417 1971 raft_consensus.cc:1942] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is already servicing an update from a current leader or another vote. I0819 06:33:49.306180 2094 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:49.307257 1781 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Service unavailable: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is already servicing an update from a current leader or another vote. I0819 06:33:49.308544 1779 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6f8e517e75d440519fb9d468724d7798. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:49.308785 1779 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0819 06:33:49.317018 2209 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.025s I0819 06:33:49.317306 2209 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority W0819 06:33:51.749737 2184 log.cc:665] Time spent T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Append to log took a long time: real 0.187s user 0.000s sys 0.000s W0819 06:33:51.749832 2198 log.cc:665] Time spent T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Append to log took a long time: real 0.210s user 0.000s sys 0.000s W0819 06:33:51.750418 2182 log.cc:665] Time spent T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b: Append to log took a long time: real 0.214s user 0.000s sys 0.000s I0819 06:33:52.198521 2171 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:33:52.198806 2171 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.327s I0819 06:33:52.199024 2171 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:52.200057 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:52.200700 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 54716c80d5de4d238c121acf92f30ec1 I0819 06:33:52.202677 1844 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:52.203582 2028 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 54716c80d5de4d238c121acf92f30ec1. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:52.204164 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:52.208775 2031 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:52.209026 2031 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0819 06:33:52.211627 2210 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.289s I0819 06:33:52.211805 2210 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0819 06:33:52.718041 2171 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:33:52.718269 2171 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.473s I0819 06:33:52.718468 2171 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:52.719468 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:52.719863 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 54716c80d5de4d238c121acf92f30ec1 I0819 06:33:52.724597 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:52.725512 1845 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:52.726433 2028 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 54716c80d5de4d238c121acf92f30ec1. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:52.730141 2031 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:52.730393 2031 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0819 06:33:52.733147 2212 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.452s I0819 06:33:52.733427 2212 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0819 06:33:52.824453 2164 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:33:52.824745 2164 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.204s I0819 06:33:52.824975 2164 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:52.825949 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:52.826328 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6f8e517e75d440519fb9d468724d7798 I0819 06:33:52.827842 2091 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:52.829285 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:52.830350 1781 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:52.830860 1779 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6f8e517e75d440519fb9d468724d7798. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:52.831082 1779 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0819 06:33:52.835933 2213 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.048s I0819 06:33:52.836299 2213 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0819 06:33:53.167534 2164 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:33:53.167817 2164 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.370s I0819 06:33:53.167991 2164 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:53.168756 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:53.169147 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6f8e517e75d440519fb9d468724d7798 I0819 06:33:53.171463 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:53.172191 2094 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:53.173928 1781 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:53.174484 1779 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6f8e517e75d440519fb9d468724d7798. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:53.174665 1779 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0819 06:33:53.174995 2213 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.299s I0819 06:33:53.175186 2213 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0819 06:33:53.303047 2171 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:33:53.303354 2171 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.585s I0819 06:33:53.303591 2171 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:53.304553 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:53.304945 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 54716c80d5de4d238c121acf92f30ec1 I0819 06:33:53.306429 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:53.307852 1844 raft_consensus.cc:1942] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is already servicing an update from a current leader or another vote. I0819 06:33:53.308758 2031 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:53.309018 2028 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 54716c80d5de4d238c121acf92f30ec1. Message: Service unavailable: T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is already servicing an update from a current leader or another vote. I0819 06:33:53.309242 2028 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0819 06:33:53.312477 2215 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.402s I0819 06:33:53.312738 2215 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0819 06:33:53.832000 2171 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:33:53.832286 2171 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.156s I0819 06:33:53.832664 2171 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:53.833659 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:53.834054 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 54716c80d5de4d238c121acf92f30ec1 I0819 06:33:53.838032 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:53.838251 1844 raft_consensus.cc:1942] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is already servicing an update from a current leader or another vote. I0819 06:33:53.839119 2031 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:53.842242 2028 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 54716c80d5de4d238c121acf92f30ec1. Message: Service unavailable: T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is already servicing an update from a current leader or another vote. I0819 06:33:53.842475 2028 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0819 06:33:53.845508 2216 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.902s I0819 06:33:53.845780 2216 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0819 06:33:54.167012 2171 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:33:54.167258 2171 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.282s I0819 06:33:54.167424 2171 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:54.168220 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:54.168573 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 54716c80d5de4d238c121acf92f30ec1 I0819 06:33:54.169867 1845 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:54.170615 2028 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 54716c80d5de4d238c121acf92f30ec1. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:54.172101 1971 raft_consensus.cc:1942] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is already servicing an update from a current leader or another vote. I0819 06:33:54.172986 2031 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Service unavailable: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is already servicing an update from a current leader or another vote. I0819 06:33:54.173219 2031 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0819 06:33:54.174759 2216 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 1.134s I0819 06:33:54.174953 2216 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0819 06:33:54.716110 2171 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:33:54.716434 2171 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.282s I0819 06:33:54.716657 2171 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:54.717684 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:54.718092 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 54716c80d5de4d238c121acf92f30ec1 I0819 06:33:54.719629 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:54.721817 1845 raft_consensus.cc:1942] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is already servicing an update from a current leader or another vote. I0819 06:33:54.722946 2028 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 54716c80d5de4d238c121acf92f30ec1. Message: Service unavailable: T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is already servicing an update from a current leader or another vote. I0819 06:33:54.724035 2031 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:54.724295 2031 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0819 06:33:54.727599 2217 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 1.275s I0819 06:33:54.727850 2217 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0819 06:33:55.833811 2164 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:33:55.834098 2164 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.120s I0819 06:33:55.834313 2164 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:55.835336 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:55.835633 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6f8e517e75d440519fb9d468724d7798 I0819 06:33:55.836838 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:55.837267 2094 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:55.837975 1781 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:55.838227 1779 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6f8e517e75d440519fb9d468724d7798. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:55.838429 1779 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0819 06:33:55.851666 2218 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.122s I0819 06:33:55.851969 2218 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0819 06:33:56.308517 2164 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:33:56.308791 2164 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.392s I0819 06:33:56.309006 2164 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:56.309984 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:56.310364 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6f8e517e75d440519fb9d468724d7798 I0819 06:33:56.311589 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:56.312625 1781 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:56.320750 2091 raft_consensus.cc:1942] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is already servicing an update from a current leader or another vote. I0819 06:33:56.323446 1779 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6f8e517e75d440519fb9d468724d7798. Message: Service unavailable: T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is already servicing an update from a current leader or another vote. I0819 06:33:56.323693 1779 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0819 06:33:56.324455 2218 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.117s I0819 06:33:56.324648 2218 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0819 06:33:57.170744 2171 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:33:57.170972 2171 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 3.202s I0819 06:33:57.171149 2171 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:57.171923 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:57.172322 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 54716c80d5de4d238c121acf92f30ec1 I0819 06:33:57.173867 1844 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:57.174868 2028 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 54716c80d5de4d238c121acf92f30ec1. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:57.176471 1971 raft_consensus.cc:1942] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is already servicing an update from a current leader or another vote. I0819 06:33:57.178767 2031 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Service unavailable: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is already servicing an update from a current leader or another vote. I0819 06:33:57.179008 2031 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0819 06:33:57.186766 2219 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.949s I0819 06:33:57.187063 2219 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0819 06:33:57.331182 2164 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:33:57.331457 2164 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.855s I0819 06:33:57.331670 2164 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:57.332684 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:57.333073 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6f8e517e75d440519fb9d468724d7798 I0819 06:33:57.337174 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:57.339629 1781 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:57.346243 2091 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:57.347306 1779 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6f8e517e75d440519fb9d468724d7798. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:57.347587 1779 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0819 06:33:57.350056 2220 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.714s I0819 06:33:57.350320 2220 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0819 06:33:57.829043 2164 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:33:57.829322 2164 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 1.725s I0819 06:33:57.829509 2164 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:57.830257 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:57.831447 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6f8e517e75d440519fb9d468724d7798 I0819 06:33:57.834915 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:57.836751 1781 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:57.839571 2091 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:57.840544 1779 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6f8e517e75d440519fb9d468724d7798. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:57.840729 1779 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0819 06:33:57.841295 2220 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.253s I0819 06:33:57.841495 2220 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0819 06:33:58.168998 2171 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:33:58.169286 2171 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 4.446s I0819 06:33:58.169530 2171 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:58.170523 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:58.170909 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 54716c80d5de4d238c121acf92f30ec1 I0819 06:33:58.175910 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:58.176795 1845 raft_consensus.cc:1942] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is already servicing an update from a current leader or another vote. I0819 06:33:58.177448 2031 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:58.177745 2028 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 54716c80d5de4d238c121acf92f30ec1. Message: Service unavailable: T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is already servicing an update from a current leader or another vote. I0819 06:33:58.177985 2028 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0819 06:33:58.179250 2221 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.892s I0819 06:33:58.179497 2221 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0819 06:33:58.805793 2164 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:33:58.806021 2164 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 1.748s I0819 06:33:58.806198 2164 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:58.807114 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:58.807425 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6f8e517e75d440519fb9d468724d7798 I0819 06:33:58.808657 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:58.809211 2094 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:58.810329 1779 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6f8e517e75d440519fb9d468724d7798. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:58.812149 1781 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:58.813158 1781 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0819 06:33:58.838999 2222 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 1.622s I0819 06:33:58.839308 2222 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0819 06:33:59.305791 2171 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:33:59.306010 2171 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 3.803s I0819 06:33:59.306183 2171 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:33:59.306928 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:33:59.307245 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 54716c80d5de4d238c121acf92f30ec1 I0819 06:33:59.308733 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:59.309039 1845 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:59.311645 2028 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 54716c80d5de4d238c121acf92f30ec1. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:59.312011 2031 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:33:59.312242 2031 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0819 06:33:59.325202 2223 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 2.788s I0819 06:33:59.325428 2223 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0819 06:34:00.172833 2171 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:34:00.173421 2171 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 5.226s I0819 06:34:00.173643 2171 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:34:00.174623 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:34:00.175020 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 54716c80d5de4d238c121acf92f30ec1 I0819 06:34:00.178165 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:34:00.185245 2031 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:34:00.185420 1844 raft_consensus.cc:1942] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is already servicing an update from a current leader or another vote. I0819 06:34:00.186625 2028 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 54716c80d5de4d238c121acf92f30ec1. Message: Service unavailable: T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is already servicing an update from a current leader or another vote. I0819 06:34:00.186862 2028 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0819 06:34:00.201203 2224 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 12.068s I0819 06:34:00.202064 2224 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0819 06:34:00.310590 2164 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:34:00.310878 2164 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 3.760s I0819 06:34:00.311095 2164 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:34:00.312103 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:34:00.312490 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6f8e517e75d440519fb9d468724d7798 I0819 06:34:00.313652 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:34:00.313987 2094 raft_consensus.cc:1942] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is already servicing an update from a current leader or another vote. I0819 06:34:00.314726 1781 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is either leader or believes a valid leader to be alive. I0819 06:34:00.315752 1779 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6f8e517e75d440519fb9d468724d7798. Message: Service unavailable: T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica is already servicing an update from a current leader or another vote. I0819 06:34:00.315970 1779 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0819 06:34:00.334712 2225 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 1.155s I0819 06:34:00.335050 2225 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0819 06:34:00.716842 2171 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:34:00.717084 2171 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 10.028s I0819 06:34:00.717285 2171 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:34:00.718031 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:34:00.718387 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 54716c80d5de4d238c121acf92f30ec1 I0819 06:34:00.719748 1845 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 6.498s I0819 06:34:00.719931 1845 raft_consensus.cc:1967] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f8e517e75d440519fb9d468724d7798 in term 2. I0819 06:34:00.720319 1971 raft_consensus.cc:1942] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is already servicing an update from a current leader or another vote. I0819 06:34:00.721071 2028 leader_election.cc:349] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote granted by peer 54716c80d5de4d238c121acf92f30ec1 I0819 06:34:00.721276 2028 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. I0819 06:34:00.723551 2226 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 17.642s I0819 06:34:00.723752 2226 raft_consensus.cc:2205] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Leader pre-election won for term 3 I0819 06:34:00.723887 2226 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:34:00.724038 2226 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 9.380s I0819 06:34:00.724190 2226 raft_consensus.cc:2448] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 2 FOLLOWER]: Advancing to term 3 I0819 06:34:00.724772 2031 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Service unavailable: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is already servicing an update from a current leader or another vote. I0819 06:34:00.728884 2226 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:34:00.729787 2226 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 election: Requesting vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:34:00.729928 2094 raft_consensus.cc:929] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 3 FOLLOWER]: Rejecting Update request from peer 6d6a239df0a0404cb3455e68efcb792b for earlier term 2. Current term is 3. Ops: [] I0819 06:34:00.730206 2226 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 election: Requesting vote from peer 54716c80d5de4d238c121acf92f30ec1 I0819 06:34:00.731541 2181 consensus_queue.cc:736] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [LEADER]: Peer responded invalid term: Peer: 6f8e517e75d440519fb9d468724d7798, Is new: false, Last received: 2.2706, Next index: 2707, Last known committed idx: 2705, Last exchange result: ERROR, Needs tablet copy: false I0819 06:34:00.731853 2181 raft_consensus.cc:2443] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Stepping down as leader of term 2 I0819 06:34:00.731993 2181 raft_consensus.cc:574] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6d6a239df0a0404cb3455e68efcb792b, State: Running, Role: LEADER I0819 06:34:00.732286 2181 consensus_queue.cc:190] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 2706, Majority replicated index: 2707, Committed index: 2707, Last appended: 2.2708, Last appended by leader: 2708, Current term: 2, Majority size: -1, State: 1, Mode: NON_LEADER I0819 06:34:00.732517 2181 consensus_peers.cc:418] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b -> Peer 6f8e517e75d440519fb9d468724d7798 (127.4.126.3:53370): Closing peer: 6f8e517e75d440519fb9d468724d7798 I0819 06:34:00.732671 2181 consensus_peers.cc:418] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b -> Peer 54716c80d5de4d238c121acf92f30ec1 (127.4.126.1:33172): Closing peer: 54716c80d5de4d238c121acf92f30ec1 I0819 06:34:00.732973 2181 raft_consensus.cc:2448] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 2 FOLLOWER]: Advancing to term 3 I0819 06:34:00.734427 1844 raft_consensus.cc:1942] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is already servicing an update from a current leader or another vote. I0819 06:34:00.735482 2028 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 election: Vote denied by peer 54716c80d5de4d238c121acf92f30ec1. Message: Service unavailable: T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica is already servicing an update from a current leader or another vote. I0819 06:34:00.736325 1971 raft_consensus.cc:1914] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica has last-logged OpId of term: 2 index: 2708, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2706. I0819 06:34:00.737439 2031 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6f8e517e75d440519fb9d468724d7798 for term 3 because replica has last-logged OpId of term: 2 index: 2708, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2706. I0819 06:34:00.737709 2031 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I0819 06:34:00.738143 2226 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 11.266s I0819 06:34:00.738310 2226 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority I0819 06:34:00.753764 2164 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:34:00.754037 2164 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 1.728s I0819 06:34:00.754215 2164 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:34:00.756207 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:34:00.756616 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6f8e517e75d440519fb9d468724d7798 I0819 06:34:00.758050 1971 raft_consensus.cc:1914] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica has last-logged OpId of term: 2 index: 2708, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2707. I0819 06:34:00.758630 2094 raft_consensus.cc:1897] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 in current term 3: Already voted for candidate 6f8e517e75d440519fb9d468724d7798 in this term. I0819 06:34:00.758554 2166 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0819 06:34:00.758800 2166 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b: Snoozing failure detection for election timeout plus an additional 0.017s I0819 06:34:00.758999 2166 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:34:00.759727 1779 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6f8e517e75d440519fb9d468724d7798. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 in current term 3: Already voted for candidate 6f8e517e75d440519fb9d468724d7798 in this term. I0819 06:34:00.759943 2166 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer 54716c80d5de4d238c121acf92f30ec1 I0819 06:34:00.760288 2166 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer 6f8e517e75d440519fb9d468724d7798 I0819 06:34:00.761255 1781 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 3 because replica has last-logged OpId of term: 2 index: 2708, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2707. I0819 06:34:00.761368 1845 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 5.638s I0819 06:34:00.761484 1781 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0819 06:34:00.761597 1845 raft_consensus.cc:1967] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d6a239df0a0404cb3455e68efcb792b in term 2. I0819 06:34:00.761672 2094 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 18.181s I0819 06:34:00.761875 2094 raft_consensus.cc:1967] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d6a239df0a0404cb3455e68efcb792b in term 3. I0819 06:34:00.762140 2225 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 1.793s I0819 06:34:00.762336 2225 raft_consensus.cc:2448] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 2 FOLLOWER]: Advancing to term 3 I0819 06:34:00.762512 1903 leader_election.cc:349] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [CANDIDATE]: Term 4 pre-election: Vote granted by peer 54716c80d5de4d238c121acf92f30ec1 I0819 06:34:00.762717 1903 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. I0819 06:34:00.762930 1904 leader_election.cc:349] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [CANDIDATE]: Term 4 pre-election: Vote granted by peer 6f8e517e75d440519fb9d468724d7798 I0819 06:34:00.763252 2180 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b: Snoozing failure detection for election timeout plus an additional 0.017s I0819 06:34:00.763491 2180 raft_consensus.cc:2205] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 3 FOLLOWER]: Leader pre-election won for term 4 I0819 06:34:00.763674 2180 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0819 06:34:00.763844 2180 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b: Snoozing failure detection for election timeout plus an additional 0.015s I0819 06:34:00.763994 2180 raft_consensus.cc:2448] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 3 FOLLOWER]: Advancing to term 4 I0819 06:34:00.765544 2225 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0819 06:34:00.768254 2180 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:34:00.769255 2180 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [CANDIDATE]: Term 4 election: Requesting vote from peer 54716c80d5de4d238c121acf92f30ec1 I0819 06:34:00.769660 2180 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [CANDIDATE]: Term 4 election: Requesting vote from peer 6f8e517e75d440519fb9d468724d7798 I0819 06:34:00.770619 1845 raft_consensus.cc:2448] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 3 FOLLOWER]: Advancing to term 4 I0819 06:34:00.770979 1845 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 5.966s I0819 06:34:00.770995 2094 raft_consensus.cc:2448] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 3 FOLLOWER]: Advancing to term 4 I0819 06:34:00.771344 2094 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 18.490s I0819 06:34:00.775080 1845 raft_consensus.cc:1967] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d6a239df0a0404cb3455e68efcb792b in term 4. I0819 06:34:00.776051 1903 leader_election.cc:349] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [CANDIDATE]: Term 4 election: Vote granted by peer 54716c80d5de4d238c121acf92f30ec1 I0819 06:34:00.776262 1903 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. I0819 06:34:00.776773 2211 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b: Snoozing failure detection for election timeout plus an additional 0.016s I0819 06:34:00.776980 2211 raft_consensus.cc:2205] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 4 FOLLOWER]: Leader election won for term 4 I0819 06:34:00.777328 2211 raft_consensus.cc:540] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 4 LEADER]: Becoming Leader. State: Replica: 6d6a239df0a0404cb3455e68efcb792b, State: Running, Role: LEADER I0819 06:34:00.777637 2211 consensus_queue.cc:169] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [LEADER]: Queue going to LEADER mode. State: All replicated index: 2706, Majority replicated index: 2707, Committed index: 2707, Last appended: 2.2708, Last appended by leader: 2708, Current term: 4, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:34:00.781482 2094 raft_consensus.cc:1967] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d6a239df0a0404cb3455e68efcb792b in term 4. I0819 06:34:00.781811 1723 catalog_manager.cc:2559] T 8b79633d21dc49868650b1a34ca1c8ea reported consensus state change: term changed from 2 to 4. New consensus state: current_term: 4 leader_uuid: "6d6a239df0a0404cb3455e68efcb792b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } } I0819 06:34:00.782598 1904 leader_election.cc:349] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [CANDIDATE]: Term 4 election: Vote granted by peer 6f8e517e75d440519fb9d468724d7798 I0819 06:34:00.785109 2094 raft_consensus.cc:961] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 4 FOLLOWER]: Refusing update from remote peer 6d6a239df0a0404cb3455e68efcb792b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2706. Preceding OpId from leader: term: 4 index: 2709. (index mismatch) I0819 06:34:00.785650 1845 raft_consensus.cc:961] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Refusing update from remote peer 6d6a239df0a0404cb3455e68efcb792b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2707. Preceding OpId from leader: term: 4 index: 2709. (index mismatch) I0819 06:34:00.787436 2194 consensus_queue.cc:726] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [LEADER]: Connected to new peer: Peer: 6f8e517e75d440519fb9d468724d7798, Is new: false, Last received: 2.2706, Next index: 2707, Last known committed idx: 2705, Last exchange result: ERROR, Needs tablet copy: false I0819 06:34:00.788307 2194 consensus_queue.cc:726] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [LEADER]: Connected to new peer: Peer: 54716c80d5de4d238c121acf92f30ec1, Is new: false, Last received: 2.2707, Next index: 2708, Last known committed idx: 2707, Last exchange result: ERROR, Needs tablet copy: false I0819 06:34:01.310586 2164 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:34:01.310873 2164 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.015s I0819 06:34:01.311089 2164 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:34:01.312095 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:34:01.312866 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 6f8e517e75d440519fb9d468724d7798 I0819 06:34:01.313501 1971 raft_consensus.cc:1942] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is already servicing an update from a current leader or another vote. I0819 06:34:01.314483 2094 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:01.314646 1781 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Service unavailable: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is already servicing an update from a current leader or another vote. I0819 06:34:01.316356 1779 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 6f8e517e75d440519fb9d468724d7798. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:01.316598 1779 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I0819 06:34:01.326349 2235 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.015s I0819 06:34:01.326647 2235 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I0819 06:34:02.313223 2164 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:34:02.313508 2164 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.027s I0819 06:34:02.313720 2164 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:34:02.314733 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:34:02.315138 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 6f8e517e75d440519fb9d468724d7798 I0819 06:34:02.316315 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:02.316407 2091 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:02.317291 1779 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 6f8e517e75d440519fb9d468724d7798. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:02.318794 1781 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:02.319008 1781 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I0819 06:34:02.327847 2236 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.024s I0819 06:34:02.328140 2236 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I0819 06:34:04.310173 2164 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:34:04.310456 2164 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.031s I0819 06:34:04.310667 2164 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:34:04.311667 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:34:04.312067 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 6f8e517e75d440519fb9d468724d7798 I0819 06:34:04.317090 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:04.318150 1781 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:04.318352 2094 raft_consensus.cc:1942] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is already servicing an update from a current leader or another vote. I0819 06:34:04.319496 1779 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 6f8e517e75d440519fb9d468724d7798. Message: Service unavailable: T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is already servicing an update from a current leader or another vote. I0819 06:34:04.319741 1779 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I0819 06:34:04.333271 2237 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.022s I0819 06:34:04.334354 2237 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I0819 06:34:07.830746 2164 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:34:07.831027 2164 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.058s I0819 06:34:07.831240 2164 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:34:07.832206 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:34:07.832741 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 6f8e517e75d440519fb9d468724d7798 I0819 06:34:07.835062 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:07.835968 1781 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:07.836616 2091 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:07.837625 1779 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 6f8e517e75d440519fb9d468724d7798. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:07.837863 1779 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I0819 06:34:07.845165 2238 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.062s I0819 06:34:07.845494 2238 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I0819 06:34:08.838306 2164 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:34:08.838587 2164 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.027s I0819 06:34:08.838801 2164 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:34:08.839752 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:34:08.840124 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 6f8e517e75d440519fb9d468724d7798 I0819 06:34:08.841863 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:08.845643 1781 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:08.850843 2091 raft_consensus.cc:1942] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is already servicing an update from a current leader or another vote. I0819 06:34:08.852718 1779 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 6f8e517e75d440519fb9d468724d7798. Message: Service unavailable: T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is already servicing an update from a current leader or another vote. I0819 06:34:08.852990 1779 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I0819 06:34:08.857350 2164 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:34:08.857645 2164 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.038s I0819 06:34:08.857844 2164 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:34:08.858808 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:34:08.859206 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 6f8e517e75d440519fb9d468724d7798 I0819 06:34:08.861197 2091 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:08.861310 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:08.862401 1781 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:08.862732 1779 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 6f8e517e75d440519fb9d468724d7798. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:08.862885 1779 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I0819 06:34:08.872637 2239 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.111s I0819 06:34:08.873543 2239 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I0819 06:34:08.873950 2240 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.017s I0819 06:34:08.874207 2240 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I0819 06:34:09.190999 2164 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:34:09.191220 2164 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.254s I0819 06:34:09.191406 2164 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:34:09.192229 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:34:09.192575 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 6f8e517e75d440519fb9d468724d7798 I0819 06:34:09.193912 2091 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:09.194679 1779 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 6f8e517e75d440519fb9d468724d7798. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:09.202455 1971 raft_consensus.cc:1942] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is already servicing an update from a current leader or another vote. I0819 06:34:09.203447 1781 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Service unavailable: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is already servicing an update from a current leader or another vote. I0819 06:34:09.203685 1781 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I0819 06:34:09.204346 2239 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.089s I0819 06:34:09.204566 2239 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I0819 06:34:09.315235 2164 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:34:09.315544 2164 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.224s I0819 06:34:09.315744 2164 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:34:09.316754 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:34:09.317148 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 6f8e517e75d440519fb9d468724d7798 I0819 06:34:09.318410 2094 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:09.319140 1779 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 6f8e517e75d440519fb9d468724d7798. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:09.319779 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:09.320670 1781 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:09.320894 1781 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I0819 06:34:09.324225 2240 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.119s I0819 06:34:09.324446 2240 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I0819 06:34:09.721729 2164 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:34:09.722004 2164 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.422s I0819 06:34:09.722239 2164 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:34:09.722929 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:34:09.723246 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 6f8e517e75d440519fb9d468724d7798 I0819 06:34:09.725356 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:09.725392 2091 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:09.726347 1781 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:09.727252 1779 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 6f8e517e75d440519fb9d468724d7798. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:09.727469 1779 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I0819 06:34:09.728097 2240 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.239s I0819 06:34:09.728289 2240 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority Connected to the Master Fetched info from all 3 Tablet Servers Table test-workload is HEALTHY (1 tablet(s) checked) The metadata for 1 table(s) is HEALTHY Checksum finished in 0s: 0/3 replicas remaining (0B from disk, 60 rows summed) ----------------------- test-workload ----------------------- T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 (127.4.126.1:33172): Checksum: 38787228187 T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b (127.4.126.2:53826): Checksum: 38787228187 T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 (127.4.126.3:53370): Checksum: 38787228187 I0819 06:34:10.426301 1150 log_verifier.cc:125] Checking tablet 8b79633d21dc49868650b1a34ca1c8ea I0819 06:34:10.427115 1150 file_cache.cc:464] Constructed file cache lbm with capacity 26214 W0819 06:34:10.446362 1150 data_dirs.cc:364] IO error: Could not lock /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/data/block_manager_instance: Could not lock /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/data/block_manager_instance: lock /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/data/block_manager_instance: Resource temporarily unavailable (error 11) W0819 06:34:10.452281 1150 data_dirs.cc:365] Proceeding without lock I0819 06:34:10.495481 2268 log_block_manager.cc:2195] Read-only block manager, skipping repair I0819 06:34:10.509055 1150 fs_report.cc:345] Block manager report -------------------- 1 data directories: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0819 06:34:10.509861 1150 fs_manager.cc:280] Time spent opening block manager: real 0.064s user 0.008s sys 0.000s I0819 06:34:10.517158 1150 fs_manager.cc:283] Opened local filesystem: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data,/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/wal uuid: "54716c80d5de4d238c121acf92f30ec1" format_stamp: "Formatted at 2017-08-19 06:33:38 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:10.720073 2164 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:34:10.720381 2164 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.323s I0819 06:34:10.720618 2164 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:34:10.721616 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:34:10.722036 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 6f8e517e75d440519fb9d468724d7798 I0819 06:34:10.725322 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:10.726378 1781 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:10.726995 2094 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:10.728070 1779 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 6f8e517e75d440519fb9d468724d7798. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:10.728303 1779 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I0819 06:34:10.734473 2270 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.607s I0819 06:34:10.734771 2270 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I0819 06:34:13.401530 1150 file_cache.cc:464] Constructed file cache lbm with capacity 26214 W0819 06:34:13.411734 1150 data_dirs.cc:364] IO error: Could not lock /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/data/block_manager_instance: Could not lock /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/data/block_manager_instance: lock /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/data/block_manager_instance: Resource temporarily unavailable (error 11) W0819 06:34:13.412045 1150 data_dirs.cc:365] Proceeding without lock I0819 06:34:13.419273 2271 log_block_manager.cc:2195] Read-only block manager, skipping repair I0819 06:34:13.419622 1150 fs_report.cc:345] Block manager report -------------------- 1 data directories: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0819 06:34:13.420100 1150 fs_manager.cc:280] Time spent opening block manager: real 0.009s user 0.004s sys 0.008s I0819 06:34:13.420271 1150 fs_manager.cc:283] Opened local filesystem: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data,/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/wal uuid: "6d6a239df0a0404cb3455e68efcb792b" format_stamp: "Formatted at 2017-08-19 06:33:39 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:14.836050 2164 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:34:14.836307 2164 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.167s I0819 06:34:14.836529 2164 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } I0819 06:34:14.837438 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:34:14.837749 2164 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 6f8e517e75d440519fb9d468724d7798 I0819 06:34:14.839059 1971 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:14.839283 2094 raft_consensus.cc:1928] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:14.840283 1779 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 6f8e517e75d440519fb9d468724d7798. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:14.840684 1781 leader_election.cc:363] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 6d6a239df0a0404cb3455e68efcb792b. Message: Invalid argument: T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 54716c80d5de4d238c121acf92f30ec1 for term 5 because replica is either leader or believes a valid leader to be alive. I0819 06:34:14.840925 1781 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I0819 06:34:14.857447 2273 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1: Snoozing failure detection for election timeout plus an additional 0.891s I0819 06:34:14.857796 2273 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 54716c80d5de4d238c121acf92f30ec1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I0819 06:34:15.237258 1150 file_cache.cc:464] Constructed file cache lbm with capacity 26214 W0819 06:34:15.244510 1150 data_dirs.cc:364] IO error: Could not lock /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/data/block_manager_instance: Could not lock /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/data/block_manager_instance: lock /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/data/block_manager_instance: Resource temporarily unavailable (error 11) W0819 06:34:15.244683 1150 data_dirs.cc:365] Proceeding without lock I0819 06:34:15.247613 2274 log_block_manager.cc:2195] Read-only block manager, skipping repair I0819 06:34:15.247891 1150 fs_report.cc:345] Block manager report -------------------- 1 data directories: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0819 06:34:15.248176 1150 fs_manager.cc:280] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s I0819 06:34:15.248355 1150 fs_manager.cc:283] Opened local filesystem: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data,/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestChurnyElections_WithDuplicateKeys.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/wal uuid: "6f8e517e75d440519fb9d468724d7798" format_stamp: "Formatted at 2017-08-19 06:33:39 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:16.650719 1150 log_verifier.cc:178] Verified matching terms for 3936 ops in tablet 8b79633d21dc49868650b1a34ca1c8ea ================== WARNING: ThreadSanitizer: data race (pid=1150) Write of size 8 at 0x7b2c00002cb8 by main thread: #0 operator delete(void*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:73 (raft_consensus-itest+0x502d61) #1 std::__1::default_delete::operator()(kudu::ThreadPoolToken*) const /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/c++/v1/memory:2397:13 (libkudu_util.so+0x1ce983) #2 std::__1::unique_ptr >::reset(kudu::ThreadPoolToken*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/c++/v1/memory:2603 (libkudu_util.so+0x1ce983) #3 std::__1::unique_ptr >::~unique_ptr() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/c++/v1/memory:2571 (libkudu_util.so+0x1ce983) #4 kudu::ThreadPool::~ThreadPool() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:339 (libkudu_util.so+0x1ce983) #5 kudu::DefaultDeleter::operator()(kudu::ThreadPool*) const /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/gscoped_ptr.h:145:5 (libmaster.so+0xc274e) #6 kudu::internal::gscoped_ptr_impl >::~gscoped_ptr_impl() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/gscoped_ptr.h:228:7 (libmaster.so+0xc2719) #7 gscoped_ptr >::~gscoped_ptr() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/gscoped_ptr.h:318:7 (libmaster.so+0xb3789) #8 kudu::DnsResolver::~DnsResolver() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/net/dns_resolver.cc:45:1 (libkudu_util.so+0x19274a) #9 kudu::DefaultDeleter::operator()(kudu::DnsResolver*) const /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/gscoped_ptr.h:145:5 (libkudu_client.so+0xd0dae) #10 kudu::internal::gscoped_ptr_impl >::reset(kudu::DnsResolver*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/gscoped_ptr.h:254:7 (libkudu_client.so+0xd0d64) #11 gscoped_ptr >::reset(kudu::DnsResolver*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/gscoped_ptr.h:375:46 (libkudu_client.so+0xc5f90) #12 kudu::client::KuduClient::Data::~Data() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/client-internal.cc:343:17 (libkudu_client.so+0xd6add) #13 kudu::client::KuduClient::~KuduClient() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/client.cc:334:3 (libkudu_client.so+0xbbfac) #14 std::__1::default_delete::operator()(kudu::client::KuduClient*) const /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/c++/v1/memory:2397:13 (libkudu_client.so+0xd067b) #15 std::__1::__shared_ptr_pointer, std::__1::allocator >::__on_zero_shared() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/c++/v1/memory:3795 (libkudu_client.so+0xd067b) #16 __release_shared /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-4.0.0.src/projects/libcxx/src/memory.cpp:67:9 (libc++.so.1+0xc095d) #17 std::__1::__shared_weak_count::__release_shared() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-4.0.0.src/projects/libcxx/src/memory.cpp:92 (libc++.so.1+0xc095d) #18 std::__1::shared_ptr::~shared_ptr() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/c++/v1/memory:4626:19 (raft_consensus-itest+0x549038) #19 kudu::TestWorkload::~TestWorkload() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/test_workload.cc:78:1 (libintegration-tests.so+0x867d3) #20 kudu::tserver::RaftConsensusITest_TestChurnyElections_WithDuplicateKeys_Test::TestBody() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/raft_consensus-itest.cc:1043:1 (raft_consensus-itest+0x50f9b9) #21 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x52b39) #22 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x52b39) #23 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474:5 (libgmock.so+0x32b97) #24 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x34036) #25 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x34da6) #26 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x40896) #27 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x53a19) #28 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x53a19) #29 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x402e1) #30 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x33db) #31 main /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/test_main.cc:104:13 (libkudu_test_main.so+0x2b96) Previous read of size 8 at 0x7b2c00002cb8 by thread T17: #0 scoped_refptr::operator kudu::Histogram* scoped_refptr::*() const /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/ref_counted.h:269:38 (libkrpc.so+0xa34b9) #1 kudu::ThreadPool::DoSubmit(std::__1::shared_ptr, kudu::ThreadPoolToken*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:534:7 (libkudu_util.so+0x1cd7f9) #2 kudu::ThreadPool::Submit(std::__1::shared_ptr) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:452:10 (libkudu_util.so+0x1cf42f) #3 kudu::ThreadPool::SubmitFunc(boost::function) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:448:10 (libkudu_util.so+0x1cf4c9) #4 kudu::DnsResolver::ResolveAddresses(kudu::HostPort const&, std::__1::vector >*, kudu::Callback const&) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/net/dns_resolver.cc:57:21 (libkudu_util.so+0x1927fb) #5 kudu::client::internal::RemoteTabletServer::InitProxy(kudu::client::KuduClient*, kudu::Callback const&) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/meta_cache.cc:135:33 (libkudu_client.so+0xf8c53) #6 kudu::client::internal::MetaCacheServerPicker::PickLeader(kudu::Callback const&, kudu::MonoTime const&) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/meta_cache.cc:443:11 (libkudu_client.so+0xfaf0f) #7 kudu::rpc::RetriableRpc::SendRpc() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/retriable_rpc.h:144:19 (libkudu_client.so+0xb2a98) #8 kudu::rpc::RpcRetrier::DelayedRetryCb(kudu::rpc::Rpc*, kudu::Status const&) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/rpc.cc:90:10 (libkrpc.so+0xd4b0c) #9 boost::_mfi::mf2::operator()(kudu::rpc::RpcRetrier*, kudu::rpc::Rpc*, kudu::Status const&) const /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/bind/mem_fn_template.hpp:280:29 (libkrpc.so+0xd54f7) #10 void boost::_bi::list3, boost::_bi::value, boost::arg<1> >::operator(), boost::_bi::rrlist1 >(boost::_bi::type, boost::_mfi::mf2&, boost::_bi::rrlist1&, int) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/bind/bind.hpp:398:9 (libkrpc.so+0xd5432) #11 void boost::_bi::bind_t, boost::_bi::list3, boost::_bi::value, boost::arg<1> > >::operator()(kudu::Status const&) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/bind/bind.hpp:1234:16 (libkrpc.so+0xd5358) #12 boost::detail::function::void_function_obj_invoker1, boost::_bi::list3, boost::_bi::value, boost::arg<1> > >, void, kudu::Status const&>::invoke(boost::detail::function::function_buffer&, kudu::Status const&) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xd50f3) #13 boost::function1::operator()(kudu::Status const&) const /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xc3234) #14 kudu::rpc::DelayedTask::TimerHandler(ev::timer&, int) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/reactor.cc:628:5 (libkrpc.so+0xc1074) #15 void ev::base::method_thunk(ev_loop*, ev_timer*, int) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/ev++.h:479:7 (libkrpc.so+0xc7d1a) #16 ev_invoke_pending /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/libev-4.20/ev.c:3155:11 (libev.so.4+0x9800) #17 ev_run /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/libev-4.20/ev.c:3555:7 (libev.so.4+0xa90d) #18 ev::loop_ref::run(int) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/ev++.h:211:7 (libkrpc.so+0xc2b08) #19 kudu::rpc::ReactorThread::RunThread() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/reactor.cc:395:9 (libkrpc.so+0xbd46a) #20 boost::_mfi::mf0::operator()(kudu::rpc::ReactorThread*) const /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/bind/mem_fn_template.hpp:49:29 (libkrpc.so+0xc5f86) #21 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/bind/bind.hpp:259:9 (libkrpc.so+0xc5eda) #22 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/bind/bind.hpp:1222:16 (libkrpc.so+0xc5e63) #23 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xc5c69) #24 boost::function0::operator()() const /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xb7251) #25 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.cc:595:3 (libkudu_util.so+0x1c76fe) As if synchronized via sleep: #0 nanosleep /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:357 (raft_consensus-itest+0x49e1e7) #1 boost::detail::yield(unsigned int) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/smart_ptr/detail/yield_k.hpp:151:9 (libmaster.so+0xd3ab6) #2 kudu::Thread::StartThread(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, boost::function const&, unsigned long, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.cc:549:7 (libkudu_util.so+0x1c70e1) #3 kudu::Status kudu::Thread::Create(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, void (kudu::ThreadPool::* const&)(bool), kudu::ThreadPool* const&, bool const&, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.h:164:12 (libkudu_util.so+0x1d1736) #4 kudu::ThreadPool::CreateThreadUnlocked() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:694:14 (libkudu_util.so+0x1cf111) #5 kudu::ThreadPool::DoSubmit(std::__1::shared_ptr, kudu::ThreadPoolToken*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:491:21 (libkudu_util.so+0x1cd5d1) #6 kudu::ThreadPool::Submit(std::__1::shared_ptr) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:452:10 (libkudu_util.so+0x1cf42f) #7 kudu::ThreadPool::SubmitFunc(boost::function) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:448:10 (libkudu_util.so+0x1cf4c9) #8 kudu::DnsResolver::ResolveAddresses(kudu::HostPort const&, std::__1::vector >*, kudu::Callback const&) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/net/dns_resolver.cc:57:21 (libkudu_util.so+0x1927fb) #9 kudu::client::internal::RemoteTabletServer::InitProxy(kudu::client::KuduClient*, kudu::Callback const&) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/meta_cache.cc:135:33 (libkudu_client.so+0xf8c53) #10 kudu::client::KuduClient::Data::GetTabletServer(kudu::client::KuduClient*, scoped_refptr const&, kudu::client::KuduClient::ReplicaSelection, std::__1::set, std::__1::allocator >, std::__1::less, std::__1::allocator > >, std::__1::allocator, std::__1::allocator > > > const&, std::__1::vector >*, kudu::client::internal::RemoteTabletServer**) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/client-internal.cc:424:8 (libkudu_client.so+0xd71b9) #11 kudu::client::KuduScanner::Data::OpenTablet(std::__1::basic_string, std::__1::allocator > const&, kudu::MonoTime const&, std::__1::set, std::__1::allocator >, std::__1::less, std::__1::allocator > >, std::__1::allocator, std::__1::allocator > > >*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/scanner-internal.cc:388:53 (libkudu_client.so+0x11ca83) #12 kudu::client::KuduScanner::Data::OpenNextTablet(kudu::MonoTime const&, std::__1::set, std::__1::allocator >, std::__1::less, std::__1::allocator > >, std::__1::allocator, std::__1::allocator > > >*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/scanner-internal.cc:243:10 (libkudu_client.so+0x11c23f) #13 kudu::client::KuduScanner::Open() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/client.cc:1331:3 (libkudu_client.so+0xc283d) #14 kudu::ClusterVerifier::DoCheckRowCount(std::__1::basic_string, std::__1::allocator > const&, kudu::ClusterVerifier::ComparisonMode, int) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/cluster_verifier.cc:138:3 (libintegration-tests.so+0x57ad4) #15 kudu::ClusterVerifier::CheckRowCount(std::__1::basic_string, std::__1::allocator > const&, kudu::ClusterVerifier::ComparisonMode, int) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/cluster_verifier.cc:116:3 (libintegration-tests.so+0x57758) #16 kudu::tserver::RaftConsensusITest::DoTestChurnyElections(kudu::TestWorkload*, int) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/raft_consensus-itest.cc:1001:3 (raft_consensus-itest+0x50f417) #17 kudu::tserver::RaftConsensusITest_TestChurnyElections_WithDuplicateKeys_Test::TestBody() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/raft_consensus-itest.cc:1042:3 (raft_consensus-itest+0x50f9ad) #18 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x52b39) #19 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x52b39) #20 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474:5 (libgmock.so+0x32b97) #21 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x34036) #22 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x34da6) #23 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x40896) #24 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x53a19) #25 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x53a19) #26 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x402e1) #27 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x33db) #28 main /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/test_main.cc:104:13 (libkudu_test_main.so+0x2b96) Thread T17 'rpc reactor-214' (tid=2145, running) created by main thread at: #0 pthread_create /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:897 (raft_consensus-itest+0x4a28d3) #1 kudu::Thread::StartThread(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, boost::function const&, unsigned long, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.cc:518:15 (libkudu_util.so+0x1c6ef7) #2 kudu::Status kudu::Thread::Create(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, void (kudu::rpc::ReactorThread::* const&)(), kudu::rpc::ReactorThread* const&, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.h:158:12 (libkrpc.so+0xc23a5) #3 kudu::rpc::ReactorThread::Init() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/reactor.cc:145:10 (libkrpc.so+0xbd2c5) #4 kudu::rpc::Reactor::Init() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/reactor.cc:645:18 (libkrpc.so+0xc14a1) #5 kudu::rpc::Messenger::Init() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/messenger.cc:517:5 (libkrpc.so+0xa6572) #6 kudu::rpc::MessengerBuilder::Build(std::__1::shared_ptr*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/messenger.cc:283:3 (libkrpc.so+0xa5b65) #7 kudu::client::KuduClientBuilder::Build(std::__1::shared_ptr*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/client.cc:295:3 (libkudu_client.so+0xbb51f) #8 kudu::ExternalMiniCluster::CreateClient(kudu::client::KuduClientBuilder*, std::__1::shared_ptr*) const /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/external_mini_cluster.cc:618:19 (libintegration-tests.so+0x64171) #9 kudu::TestWorkload::Setup() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/test_workload.cc:231:5 (libintegration-tests.so+0x881fc) #10 kudu::tserver::RaftConsensusITest::DoTestChurnyElections(kudu::TestWorkload*, int) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/raft_consensus-itest.cc:981:13 (raft_consensus-itest+0x50f0ee) #11 kudu::tserver::RaftConsensusITest_TestChurnyElections_WithDuplicateKeys_Test::TestBody() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/raft_consensus-itest.cc:1042:3 (raft_consensus-itest+0x50f9ad) #12 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x52b39) #13 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x52b39) #14 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474:5 (libgmock.so+0x32b97) #15 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x34036) #16 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x34da6) #17 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x40896) #18 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x53a19) #19 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x53a19) #20 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x402e1) #21 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x33db) #22 main /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/test_main.cc:104:13 (libkudu_test_main.so+0x2b96) SUMMARY: ThreadSanitizer: data race /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:73 in operator delete(void*) ================== ================== WARNING: ThreadSanitizer: data race (pid=1150) Write of size 8 at 0x7b5000005cf0 by main thread: #0 pthread_cond_destroy /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1102 (raft_consensus-itest+0x4a1b5c) #1 kudu::ConditionVariable::~ConditionVariable() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/condition_variable.cc:57:12 (libkudu_util.so+0xf77fe) #2 kudu::ThreadPool::~ThreadPool() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:339:1 (libkudu_util.so+0x1ce9b3) #3 kudu::DefaultDeleter::operator()(kudu::ThreadPool*) const /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/gscoped_ptr.h:145:5 (libmaster.so+0xc274e) #4 kudu::internal::gscoped_ptr_impl >::~gscoped_ptr_impl() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/gscoped_ptr.h:228:7 (libmaster.so+0xc2719) #5 gscoped_ptr >::~gscoped_ptr() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/gscoped_ptr.h:318:7 (libmaster.so+0xb3789) #6 kudu::DnsResolver::~DnsResolver() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/net/dns_resolver.cc:45:1 (libkudu_util.so+0x19274a) #7 kudu::DefaultDeleter::operator()(kudu::DnsResolver*) const /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/gscoped_ptr.h:145:5 (libkudu_client.so+0xd0dae) #8 kudu::internal::gscoped_ptr_impl >::reset(kudu::DnsResolver*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/gscoped_ptr.h:254:7 (libkudu_client.so+0xd0d64) #9 gscoped_ptr >::reset(kudu::DnsResolver*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/gscoped_ptr.h:375:46 (libkudu_client.so+0xc5f90) #10 kudu::client::KuduClient::Data::~Data() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/client-internal.cc:343:17 (libkudu_client.so+0xd6add) #11 kudu::client::KuduClient::~KuduClient() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/client.cc:334:3 (libkudu_client.so+0xbbfac) #12 std::__1::default_delete::operator()(kudu::client::KuduClient*) const /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/c++/v1/memory:2397:13 (libkudu_client.so+0xd067b) #13 std::__1::__shared_ptr_pointer, std::__1::allocator >::__on_zero_shared() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/c++/v1/memory:3795 (libkudu_client.so+0xd067b) #14 __release_shared /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-4.0.0.src/projects/libcxx/src/memory.cpp:67:9 (libc++.so.1+0xc095d) #15 std::__1::__shared_weak_count::__release_shared() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-4.0.0.src/projects/libcxx/src/memory.cpp:92 (libc++.so.1+0xc095d) #16 std::__1::shared_ptr::~shared_ptr() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/c++/v1/memory:4626:19 (raft_consensus-itest+0x549038) #17 kudu::TestWorkload::~TestWorkload() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/test_workload.cc:78:1 (libintegration-tests.so+0x867d3) #18 kudu::tserver::RaftConsensusITest_TestChurnyElections_WithDuplicateKeys_Test::TestBody() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/raft_consensus-itest.cc:1043:1 (raft_consensus-itest+0x50f9b9) #19 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x52b39) #20 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x52b39) #21 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474:5 (libgmock.so+0x32b97) #22 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x34036) #23 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x34da6) #24 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x40896) #25 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x53a19) #26 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x53a19) #27 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x402e1) #28 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x33db) #29 main /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/test_main.cc:104:13 (libkudu_test_main.so+0x2b96) Previous read of size 8 at 0x7b5000005cf0 by thread T17: #0 pthread_cond_signal /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1088 (raft_consensus-itest+0x4a1890) #1 kudu::ConditionVariable::Signal() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/condition_variable.cc:136:12 (libkudu_util.so+0xf7c7e) #2 kudu::ThreadPool::DoSubmit(std::__1::shared_ptr, kudu::ThreadPoolToken*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:529:14 (libkudu_util.so+0x1cd7bb) #3 kudu::ThreadPool::Submit(std::__1::shared_ptr) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:452:10 (libkudu_util.so+0x1cf42f) #4 kudu::ThreadPool::SubmitFunc(boost::function) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:448:10 (libkudu_util.so+0x1cf4c9) #5 kudu::DnsResolver::ResolveAddresses(kudu::HostPort const&, std::__1::vector >*, kudu::Callback const&) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/net/dns_resolver.cc:57:21 (libkudu_util.so+0x1927fb) #6 kudu::client::internal::RemoteTabletServer::InitProxy(kudu::client::KuduClient*, kudu::Callback const&) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/meta_cache.cc:135:33 (libkudu_client.so+0xf8c53) #7 kudu::client::internal::MetaCacheServerPicker::PickLeader(kudu::Callback const&, kudu::MonoTime const&) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/meta_cache.cc:443:11 (libkudu_client.so+0xfaf0f) #8 kudu::rpc::RetriableRpc::SendRpc() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/retriable_rpc.h:144:19 (libkudu_client.so+0xb2a98) #9 kudu::rpc::RpcRetrier::DelayedRetryCb(kudu::rpc::Rpc*, kudu::Status const&) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/rpc.cc:90:10 (libkrpc.so+0xd4b0c) #10 boost::_mfi::mf2::operator()(kudu::rpc::RpcRetrier*, kudu::rpc::Rpc*, kudu::Status const&) const /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/bind/mem_fn_template.hpp:280:29 (libkrpc.so+0xd54f7) #11 void boost::_bi::list3, boost::_bi::value, boost::arg<1> >::operator(), boost::_bi::rrlist1 >(boost::_bi::type, boost::_mfi::mf2&, boost::_bi::rrlist1&, int) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/bind/bind.hpp:398:9 (libkrpc.so+0xd5432) #12 void boost::_bi::bind_t, boost::_bi::list3, boost::_bi::value, boost::arg<1> > >::operator()(kudu::Status const&) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/bind/bind.hpp:1234:16 (libkrpc.so+0xd5358) #13 boost::detail::function::void_function_obj_invoker1, boost::_bi::list3, boost::_bi::value, boost::arg<1> > >, void, kudu::Status const&>::invoke(boost::detail::function::function_buffer&, kudu::Status const&) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xd50f3) #14 boost::function1::operator()(kudu::Status const&) const /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xc3234) #15 kudu::rpc::DelayedTask::TimerHandler(ev::timer&, int) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/reactor.cc:628:5 (libkrpc.so+0xc1074) #16 void ev::base::method_thunk(ev_loop*, ev_timer*, int) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/ev++.h:479:7 (libkrpc.so+0xc7d1a) #17 ev_invoke_pending /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/libev-4.20/ev.c:3155:11 (libev.so.4+0x9800) #18 ev_run /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/libev-4.20/ev.c:3555:7 (libev.so.4+0xa90d) #19 ev::loop_ref::run(int) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/ev++.h:211:7 (libkrpc.so+0xc2b08) #20 kudu::rpc::ReactorThread::RunThread() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/reactor.cc:395:9 (libkrpc.so+0xbd46a) #21 boost::_mfi::mf0::operator()(kudu::rpc::ReactorThread*) const /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/bind/mem_fn_template.hpp:49:29 (libkrpc.so+0xc5f86) #22 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/bind/bind.hpp:259:9 (libkrpc.so+0xc5eda) #23 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/bind/bind.hpp:1222:16 (libkrpc.so+0xc5e63) #24 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xc5c69) #25 boost::function0::operator()() const /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xb7251) #26 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.cc:595:3 (libkudu_util.so+0x1c76fe) As if synchronized via sleep: #0 nanosleep /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:357 (raft_consensus-itest+0x49e1e7) #1 boost::detail::yield(unsigned int) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/smart_ptr/detail/yield_k.hpp:151:9 (libmaster.so+0xd3ab6) #2 kudu::Thread::StartThread(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, boost::function const&, unsigned long, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.cc:549:7 (libkudu_util.so+0x1c70e1) #3 kudu::Status kudu::Thread::Create(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, void (kudu::ThreadPool::* const&)(bool), kudu::ThreadPool* const&, bool const&, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.h:164:12 (libkudu_util.so+0x1d1736) #4 kudu::ThreadPool::CreateThreadUnlocked() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:694:14 (libkudu_util.so+0x1cf111) #5 kudu::ThreadPool::DoSubmit(std::__1::shared_ptr, kudu::ThreadPoolToken*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:491:21 (libkudu_util.so+0x1cd5d1) #6 kudu::ThreadPool::Submit(std::__1::shared_ptr) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:452:10 (libkudu_util.so+0x1cf42f) #7 kudu::ThreadPool::SubmitFunc(boost::function) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:448:10 (libkudu_util.so+0x1cf4c9) #8 kudu::DnsResolver::ResolveAddresses(kudu::HostPort const&, std::__1::vector >*, kudu::Callback const&) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/net/dns_resolver.cc:57:21 (libkudu_util.so+0x1927fb) #9 kudu::client::internal::RemoteTabletServer::InitProxy(kudu::client::KuduClient*, kudu::Callback const&) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/meta_cache.cc:135:33 (libkudu_client.so+0xf8c53) #10 kudu::client::KuduClient::Data::GetTabletServer(kudu::client::KuduClient*, scoped_refptr const&, kudu::client::KuduClient::ReplicaSelection, std::__1::set, std::__1::allocator >, std::__1::less, std::__1::allocator > >, std::__1::allocator, std::__1::allocator > > > const&, std::__1::vector >*, kudu::client::internal::RemoteTabletServer**) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/client-internal.cc:424:8 (libkudu_client.so+0xd71b9) #11 kudu::client::KuduScanner::Data::OpenTablet(std::__1::basic_string, std::__1::allocator > const&, kudu::MonoTime const&, std::__1::set, std::__1::allocator >, std::__1::less, std::__1::allocator > >, std::__1::allocator, std::__1::allocator > > >*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/scanner-internal.cc:388:53 (libkudu_client.so+0x11ca83) #12 kudu::client::KuduScanner::Data::OpenNextTablet(kudu::MonoTime const&, std::__1::set, std::__1::allocator >, std::__1::less, std::__1::allocator > >, std::__1::allocator, std::__1::allocator > > >*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/scanner-internal.cc:243:10 (libkudu_client.so+0x11c23f) #13 kudu::client::KuduScanner::Open() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/client.cc:1331:3 (libkudu_client.so+0xc283d) #14 kudu::ClusterVerifier::DoCheckRowCount(std::__1::basic_string, std::__1::allocator > const&, kudu::ClusterVerifier::ComparisonMode, int) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/cluster_verifier.cc:138:3 (libintegration-tests.so+0x57ad4) #15 kudu::ClusterVerifier::CheckRowCount(std::__1::basic_string, std::__1::allocator > const&, kudu::ClusterVerifier::ComparisonMode, int) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/cluster_verifier.cc:116:3 (libintegration-tests.so+0x57758) #16 kudu::tserver::RaftConsensusITest::DoTestChurnyElections(kudu::TestWorkload*, int) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/raft_consensus-itest.cc:1001:3 (raft_consensus-itest+0x50f417) #17 kudu::tserver::RaftConsensusITest_TestChurnyElections_WithDuplicateKeys_Test::TestBody() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/raft_consensus-itest.cc:1042:3 (raft_consensus-itest+0x50f9ad) #18 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x52b39) #19 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x52b39) #20 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474:5 (libgmock.so+0x32b97) #21 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x34036) #22 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x34da6) #23 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x40896) #24 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x53a19) #25 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x53a19) #26 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x402e1) #27 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x33db) #28 main /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/test_main.cc:104:13 (libkudu_test_main.so+0x2b96) Location is heap block of size 504 at 0x7b5000005c00 allocated by main thread: #0 operator new(unsigned long) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:41 (raft_consensus-itest+0x502ac3) #1 kudu::ThreadPoolBuilder::Build(gscoped_ptr >*) const /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:124:15 (libkudu_util.so+0x1cc8da) #2 kudu::DnsResolver::DnsResolver() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/net/dns_resolver.cc:38:3 (libkudu_util.so+0x1925c0) #3 kudu::client::KuduClientBuilder::Build(std::__1::shared_ptr*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/client.cc:315:37 (libkudu_client.so+0xbb7df) #4 kudu::ExternalMiniCluster::CreateClient(kudu::client::KuduClientBuilder*, std::__1::shared_ptr*) const /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/external_mini_cluster.cc:618:19 (libintegration-tests.so+0x64171) #5 kudu::TestWorkload::Setup() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/test_workload.cc:231:5 (libintegration-tests.so+0x881fc) #6 kudu::tserver::RaftConsensusITest::DoTestChurnyElections(kudu::TestWorkload*, int) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/raft_consensus-itest.cc:981:13 (raft_consensus-itest+0x50f0ee) #7 kudu::tserver::RaftConsensusITest_TestChurnyElections_WithDuplicateKeys_Test::TestBody() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/raft_consensus-itest.cc:1042:3 (raft_consensus-itest+0x50f9ad) #8 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x52b39) #9 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x52b39) #10 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474:5 (libgmock.so+0x32b97) #11 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x34036) #12 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x34da6) #13 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x40896) #14 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x53a19) #15 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x53a19) #16 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x402e1) #17 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x33db) #18 main /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/test_main.cc:104:13 (libkudu_test_main.so+0x2b96) Thread T17 'rpc reactor-214' (tid=2145, running) created by main thread at: #0 pthread_create /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:897 (raft_consensus-itest+0x4a28d3) #1 kudu::Thread::StartThread(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, boost::function const&, unsigned long, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.cc:518:15 (libkudu_util.so+0x1c6ef7) #2 kudu::Status kudu::Thread::Create(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, void (kudu::rpc::ReactorThread::* const&)(), kudu::rpc::ReactorThread* const&, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.h:158:12 (libkrpc.so+0xc23a5) #3 kudu::rpc::ReactorThread::Init() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/reactor.cc:145:10 (libkrpc.so+0xbd2c5) #4 kudu::rpc::Reactor::Init() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/reactor.cc:645:18 (libkrpc.so+0xc14a1) #5 kudu::rpc::Messenger::Init() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/messenger.cc:517:5 (libkrpc.so+0xa6572) #6 kudu::rpc::MessengerBuilder::Build(std::__1::shared_ptr*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/messenger.cc:283:3 (libkrpc.so+0xa5b65) #7 kudu::client::KuduClientBuilder::Build(std::__1::shared_ptr*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/client.cc:295:3 (libkudu_client.so+0xbb51f) #8 kudu::ExternalMiniCluster::CreateClient(kudu::client::KuduClientBuilder*, std::__1::shared_ptr*) const /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/external_mini_cluster.cc:618:19 (libintegration-tests.so+0x64171) #9 kudu::TestWorkload::Setup() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/test_workload.cc:231:5 (libintegration-tests.so+0x881fc) #10 kudu::tserver::RaftConsensusITest::DoTestChurnyElections(kudu::TestWorkload*, int) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/raft_consensus-itest.cc:981:13 (raft_consensus-itest+0x50f0ee) #11 kudu::tserver::RaftConsensusITest_TestChurnyElections_WithDuplicateKeys_Test::TestBody() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/raft_consensus-itest.cc:1042:3 (raft_consensus-itest+0x50f9ad) #12 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x52b39) #13 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x52b39) #14 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474:5 (libgmock.so+0x32b97) #15 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x34036) #16 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x34da6) #17 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x40896) #18 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x53a19) #19 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x53a19) #20 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x402e1) #21 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x33db) #22 main /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/test_main.cc:104:13 (libkudu_test_main.so+0x2b96) SUMMARY: ThreadSanitizer: data race /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/condition_variable.cc:57:12 in kudu::ConditionVariable::~ConditionVariable() ================== ================== WARNING: ThreadSanitizer: data race (pid=1150) Write of size 8 at 0x7b5000005dc8 by main thread: #0 operator delete(void*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:73 (raft_consensus-itest+0x502d61) #1 kudu::DefaultDeleter::operator()(kudu::ThreadPool*) const /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/gscoped_ptr.h:145:5 (libmaster.so+0xc2756) #2 kudu::internal::gscoped_ptr_impl >::~gscoped_ptr_impl() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/gscoped_ptr.h:228:7 (libmaster.so+0xc2719) #3 gscoped_ptr >::~gscoped_ptr() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/gscoped_ptr.h:318:7 (libmaster.so+0xb3789) #4 kudu::DnsResolver::~DnsResolver() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/net/dns_resolver.cc:45:1 (libkudu_util.so+0x19274a) #5 kudu::DefaultDeleter::operator()(kudu::DnsResolver*) const /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/gscoped_ptr.h:145:5 (libkudu_client.so+0xd0dae) #6 kudu::internal::gscoped_ptr_impl >::reset(kudu::DnsResolver*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/gscoped_ptr.h:254:7 (libkudu_client.so+0xd0d64) #7 gscoped_ptr >::reset(kudu::DnsResolver*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/gscoped_ptr.h:375:46 (libkudu_client.so+0xc5f90) #8 kudu::client::KuduClient::Data::~Data() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/client-internal.cc:343:17 (libkudu_client.so+0xd6add) #9 kudu::client::KuduClient::~KuduClient() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/client.cc:334:3 (libkudu_client.so+0xbbfac) #10 std::__1::default_delete::operator()(kudu::client::KuduClient*) const /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/c++/v1/memory:2397:13 (libkudu_client.so+0xd067b) #11 std::__1::__shared_ptr_pointer, std::__1::allocator >::__on_zero_shared() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/c++/v1/memory:3795 (libkudu_client.so+0xd067b) #12 __release_shared /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-4.0.0.src/projects/libcxx/src/memory.cpp:67:9 (libc++.so.1+0xc095d) #13 std::__1::__shared_weak_count::__release_shared() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-4.0.0.src/projects/libcxx/src/memory.cpp:92 (libc++.so.1+0xc095d) #14 std::__1::shared_ptr::~shared_ptr() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/c++/v1/memory:4626:19 (raft_consensus-itest+0x549038) #15 kudu::TestWorkload::~TestWorkload() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/test_workload.cc:78:1 (libintegration-tests.so+0x867d3) #16 kudu::tserver::RaftConsensusITest_TestChurnyElections_WithDuplicateKeys_Test::TestBody() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/raft_consensus-itest.cc:1043:1 (raft_consensus-itest+0x50f9b9) #17 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x52b39) #18 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x52b39) #19 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474:5 (libgmock.so+0x32b97) #20 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x34036) #21 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x34da6) #22 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x40896) #23 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x53a19) #24 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x53a19) #25 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x402e1) #26 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x33db) #27 main /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/test_main.cc:104:13 (libkudu_test_main.so+0x2b96) Previous read of size 8 at 0x7b5000005dc8 by thread T17: #0 scoped_refptr::operator kudu::Histogram* scoped_refptr::*() const /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/ref_counted.h:269:38 (libkrpc.so+0xa34b9) #1 kudu::ThreadPool::DoSubmit(std::__1::shared_ptr, kudu::ThreadPoolToken*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:531:7 (libkudu_util.so+0x1cd7ca) #2 kudu::ThreadPool::Submit(std::__1::shared_ptr) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:452:10 (libkudu_util.so+0x1cf42f) #3 kudu::ThreadPool::SubmitFunc(boost::function) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:448:10 (libkudu_util.so+0x1cf4c9) #4 kudu::DnsResolver::ResolveAddresses(kudu::HostPort const&, std::__1::vector >*, kudu::Callback const&) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/net/dns_resolver.cc:57:21 (libkudu_util.so+0x1927fb) #5 kudu::client::internal::RemoteTabletServer::InitProxy(kudu::client::KuduClient*, kudu::Callback const&) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/meta_cache.cc:135:33 (libkudu_client.so+0xf8c53) #6 kudu::client::internal::MetaCacheServerPicker::PickLeader(kudu::Callback const&, kudu::MonoTime const&) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/meta_cache.cc:443:11 (libkudu_client.so+0xfaf0f) #7 kudu::rpc::RetriableRpc::SendRpc() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/retriable_rpc.h:144:19 (libkudu_client.so+0xb2a98) #8 kudu::rpc::RpcRetrier::DelayedRetryCb(kudu::rpc::Rpc*, kudu::Status const&) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/rpc.cc:90:10 (libkrpc.so+0xd4b0c) #9 boost::_mfi::mf2::operator()(kudu::rpc::RpcRetrier*, kudu::rpc::Rpc*, kudu::Status const&) const /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/bind/mem_fn_template.hpp:280:29 (libkrpc.so+0xd54f7) #10 void boost::_bi::list3, boost::_bi::value, boost::arg<1> >::operator(), boost::_bi::rrlist1 >(boost::_bi::type, boost::_mfi::mf2&, boost::_bi::rrlist1&, int) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/bind/bind.hpp:398:9 (libkrpc.so+0xd5432) #11 void boost::_bi::bind_t, boost::_bi::list3, boost::_bi::value, boost::arg<1> > >::operator()(kudu::Status const&) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/bind/bind.hpp:1234:16 (libkrpc.so+0xd5358) #12 boost::detail::function::void_function_obj_invoker1, boost::_bi::list3, boost::_bi::value, boost::arg<1> > >, void, kudu::Status const&>::invoke(boost::detail::function::function_buffer&, kudu::Status const&) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xd50f3) #13 boost::function1::operator()(kudu::Status const&) const /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xc3234) #14 kudu::rpc::DelayedTask::TimerHandler(ev::timer&, int) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/reactor.cc:628:5 (libkrpc.so+0xc1074) #15 void ev::base::method_thunk(ev_loop*, ev_timer*, int) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/ev++.h:479:7 (libkrpc.so+0xc7d1a) #16 ev_invoke_pending /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/libev-4.20/ev.c:3155:11 (libev.so.4+0x9800) #17 ev_run /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/libev-4.20/ev.c:3555:7 (libev.so.4+0xa90d) #18 ev::loop_ref::run(int) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/ev++.h:211:7 (libkrpc.so+0xc2b08) #19 kudu::rpc::ReactorThread::RunThread() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/reactor.cc:395:9 (libkrpc.so+0xbd46a) #20 boost::_mfi::mf0::operator()(kudu::rpc::ReactorThread*) const /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/bind/mem_fn_template.hpp:49:29 (libkrpc.so+0xc5f86) #21 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/bind/bind.hpp:259:9 (libkrpc.so+0xc5eda) #22 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/bind/bind.hpp:1222:16 (libkrpc.so+0xc5e63) #23 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xc5c69) #24 boost::function0::operator()() const /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xb7251) #25 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.cc:595:3 (libkudu_util.so+0x1c76fe) As if synchronized via sleep: #0 nanosleep /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:357 (raft_consensus-itest+0x49e1e7) #1 boost::detail::yield(unsigned int) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/common/include/boost/smart_ptr/detail/yield_k.hpp:151:9 (libmaster.so+0xd3ab6) #2 kudu::Thread::StartThread(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, boost::function const&, unsigned long, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.cc:549:7 (libkudu_util.so+0x1c70e1) #3 kudu::Status kudu::Thread::Create(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, void (kudu::ThreadPool::* const&)(bool), kudu::ThreadPool* const&, bool const&, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.h:164:12 (libkudu_util.so+0x1d1736) #4 kudu::ThreadPool::CreateThreadUnlocked() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:694:14 (libkudu_util.so+0x1cf111) #5 kudu::ThreadPool::DoSubmit(std::__1::shared_ptr, kudu::ThreadPoolToken*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:491:21 (libkudu_util.so+0x1cd5d1) #6 kudu::ThreadPool::Submit(std::__1::shared_ptr) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:452:10 (libkudu_util.so+0x1cf42f) #7 kudu::ThreadPool::SubmitFunc(boost::function) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:448:10 (libkudu_util.so+0x1cf4c9) #8 kudu::DnsResolver::ResolveAddresses(kudu::HostPort const&, std::__1::vector >*, kudu::Callback const&) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/net/dns_resolver.cc:57:21 (libkudu_util.so+0x1927fb) #9 kudu::client::internal::RemoteTabletServer::InitProxy(kudu::client::KuduClient*, kudu::Callback const&) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/meta_cache.cc:135:33 (libkudu_client.so+0xf8c53) #10 kudu::client::KuduClient::Data::GetTabletServer(kudu::client::KuduClient*, scoped_refptr const&, kudu::client::KuduClient::ReplicaSelection, std::__1::set, std::__1::allocator >, std::__1::less, std::__1::allocator > >, std::__1::allocator, std::__1::allocator > > > const&, std::__1::vector >*, kudu::client::internal::RemoteTabletServer**) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/client-internal.cc:424:8 (libkudu_client.so+0xd71b9) #11 kudu::client::KuduScanner::Data::OpenTablet(std::__1::basic_string, std::__1::allocator > const&, kudu::MonoTime const&, std::__1::set, std::__1::allocator >, std::__1::less, std::__1::allocator > >, std::__1::allocator, std::__1::allocator > > >*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/scanner-internal.cc:388:53 (libkudu_client.so+0x11ca83) #12 kudu::client::KuduScanner::Data::OpenNextTablet(kudu::MonoTime const&, std::__1::set, std::__1::allocator >, std::__1::less, std::__1::allocator > >, std::__1::allocator, std::__1::allocator > > >*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/scanner-internal.cc:243:10 (libkudu_client.so+0x11c23f) #13 kudu::client::KuduScanner::Open() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/client.cc:1331:3 (libkudu_client.so+0xc283d) #14 kudu::ClusterVerifier::DoCheckRowCount(std::__1::basic_string, std::__1::allocator > const&, kudu::ClusterVerifier::ComparisonMode, int) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/cluster_verifier.cc:138:3 (libintegration-tests.so+0x57ad4) #15 kudu::ClusterVerifier::CheckRowCount(std::__1::basic_string, std::__1::allocator > const&, kudu::ClusterVerifier::ComparisonMode, int) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/cluster_verifier.cc:116:3 (libintegration-tests.so+0x57758) #16 kudu::tserver::RaftConsensusITest::DoTestChurnyElections(kudu::TestWorkload*, int) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/raft_consensus-itest.cc:1001:3 (raft_consensus-itest+0x50f417) #17 kudu::tserver::RaftConsensusITest_TestChurnyElections_WithDuplicateKeys_Test::TestBody() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/raft_consensus-itest.cc:1042:3 (raft_consensus-itest+0x50f9ad) #18 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x52b39) #19 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x52b39) #20 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474:5 (libgmock.so+0x32b97) #21 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x34036) #22 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x34da6) #23 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x40896) #24 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x53a19) #25 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x53a19) #26 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x402e1) #27 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x33db) #28 main /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/test_main.cc:104:13 (libkudu_test_main.so+0x2b96) Thread T17 'rpc reactor-214' (tid=2145, running) created by main thread at: #0 pthread_create /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:897 (raft_consensus-itest+0x4a28d3) #1 kudu::Thread::StartThread(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, boost::function const&, unsigned long, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.cc:518:15 (libkudu_util.so+0x1c6ef7) #2 kudu::Status kudu::Thread::Create(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, void (kudu::rpc::ReactorThread::* const&)(), kudu::rpc::ReactorThread* const&, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.h:158:12 (libkrpc.so+0xc23a5) #3 kudu::rpc::ReactorThread::Init() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/reactor.cc:145:10 (libkrpc.so+0xbd2c5) #4 kudu::rpc::Reactor::Init() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/reactor.cc:645:18 (libkrpc.so+0xc14a1) #5 kudu::rpc::Messenger::Init() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/messenger.cc:517:5 (libkrpc.so+0xa6572) #6 kudu::rpc::MessengerBuilder::Build(std::__1::shared_ptr*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/messenger.cc:283:3 (libkrpc.so+0xa5b65) #7 kudu::client::KuduClientBuilder::Build(std::__1::shared_ptr*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/client/client.cc:295:3 (libkudu_client.so+0xbb51f) #8 kudu::ExternalMiniCluster::CreateClient(kudu::client::KuduClientBuilder*, std::__1::shared_ptr*) const /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/external_mini_cluster.cc:618:19 (libintegration-tests.so+0x64171) #9 kudu::TestWorkload::Setup() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/test_workload.cc:231:5 (libintegration-tests.so+0x881fc) #10 kudu::tserver::RaftConsensusITest::DoTestChurnyElections(kudu::TestWorkload*, int) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/raft_consensus-itest.cc:981:13 (raft_consensus-itest+0x50f0ee) #11 kudu::tserver::RaftConsensusITest_TestChurnyElections_WithDuplicateKeys_Test::TestBody() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/raft_consensus-itest.cc:1042:3 (raft_consensus-itest+0x50f9ad) #12 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x52b39) #13 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x52b39) #14 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474:5 (libgmock.so+0x32b97) #15 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x34036) #16 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x34da6) #17 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x40896) #18 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x53a19) #19 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x53a19) #20 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x402e1) #21 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x33db) #22 main /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/test_main.cc:104:13 (libkudu_test_main.so+0x2b96) SUMMARY: ThreadSanitizer: data race /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:73 in operator delete(void*) ================== I0819 06:34:17.064626 1150 external_mini_cluster.cc:944] Killing /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver with pid 1757 W0819 06:34:17.120653 1902 connection.cc:551] server connection from 127.4.126.1:53194 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:17.120743 1903 connection.cc:551] client connection to 127.4.126.1:33172 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:17.120972 1705 connection.cc:551] client connection to 127.4.126.1:33172 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:17.121021 1903 consensus_peers.cc:396] T 8b79633d21dc49868650b1a34ca1c8ea P 6d6a239df0a0404cb3455e68efcb792b -> Peer 54716c80d5de4d238c121acf92f30ec1 (127.4.126.1:33172): Couldn't send request to peer 54716c80d5de4d238c121acf92f30ec1 for tablet 8b79633d21dc49868650b1a34ca1c8ea. Status: Network error: failed to read from TLS socket: Connection reset by peer (error 104). Retrying in the next heartbeat period. Already tried 1 times. W0819 06:34:17.121724 1903 connection.cc:190] Error closing socket: Network error: TlsSocket::Close: Broken pipe W0819 06:34:17.121836 1706 connection.cc:551] server connection from 127.4.126.1:46477 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0819 06:34:17.122018 1150 external_mini_cluster.cc:944] Killing /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver with pid 1893 W0819 06:34:17.122402 2031 connection.cc:551] server connection from 127.4.126.1:57286 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0819 06:34:17.133318 2171 raft_consensus.cc:413] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6d6a239df0a0404cb3455e68efcb792b) I0819 06:34:17.133590 2171 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.016s I0819 06:34:17.133790 2171 raft_consensus.cc:437] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6a239df0a0404cb3455e68efcb792b" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 53826 } } peers { permanent_uuid: "54716c80d5de4d238c121acf92f30ec1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 33172 } } peers { permanent_uuid: "6f8e517e75d440519fb9d468724d7798" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 53370 } } W0819 06:34:17.134203 2028 connection.cc:551] client connection to 127.4.126.1:33172 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0819 06:34:17.134745 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 6d6a239df0a0404cb3455e68efcb792b I0819 06:34:17.135115 2171 leader_election.cc:217] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 54716c80d5de4d238c121acf92f30ec1 W0819 06:34:17.141683 2028 leader_election.cc:273] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 54716c80d5de4d238c121acf92f30ec1: Network error: Client connection negotiation failed: client connection to 127.4.126.1:33172: connect: Connection refused (error 111) W0819 06:34:17.152299 2031 leader_election.cc:273] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6d6a239df0a0404cb3455e68efcb792b: Timed out: RequestConsensusVote RPC to 127.4.126.2:53826 timed out after 0.016s (SENT) I0819 06:34:17.152547 2031 leader_election.cc:244] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I0819 06:34:17.161316 2286 raft_consensus.cc:2401] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798: Snoozing failure detection for election timeout plus an additional 0.016s I0819 06:34:17.161653 2286 raft_consensus.cc:2141] T 8b79633d21dc49868650b1a34ca1c8ea P 6f8e517e75d440519fb9d468724d7798 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority W0819 06:34:17.175012 2028 connection.cc:551] server connection from 127.4.126.2:43447 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0819 06:34:17.176074 1150 external_mini_cluster.cc:944] Killing /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver with pid 2018 W0819 06:34:17.180573 1708 connection.cc:551] client connection to 127.4.126.2:53826 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:17.181226 1706 connection.cc:551] server connection from 127.4.126.2:34672 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:17.231675 1706 connection.cc:551] client connection to 127.4.126.3:53370 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0819 06:34:17.231943 1150 external_mini_cluster.cc:944] Killing /tmp/run_tha_test479zHR/build/tsan/bin/kudu-master with pid 1695 W0819 06:34:17.260614 1694 connection.cc:551] client connection to 127.4.126.254:32833 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) [ OK ] RaftConsensusITest.TestChurnyElections_WithDuplicateKeys (39532 ms) [ RUN ] RaftConsensusITest.TestLeaderStepDown I0819 06:34:17.274148 1150 test_util.cc:185] Using random seed: 1867791116 I0819 06:34:17.277310 1150 ts_itest-base.h:81] Starting cluster with: I0819 06:34:17.277412 1150 ts_itest-base.h:82] -------------- I0819 06:34:17.277504 1150 ts_itest-base.h:83] 3 tablet servers I0819 06:34:17.277592 1150 ts_itest-base.h:84] 3 replicas per TS I0819 06:34:17.277683 1150 ts_itest-base.h:85] -------------- I0819 06:34:17.279755 1150 external_mini_cluster.cc:752] Running /tmp/run_tha_test479zHR/build/tsan/bin/kudu-master /tmp/run_tha_test479zHR/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.126.254:0 --webserver_interface=127.4.126.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=flag --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0819 06:34:17.406298 2292 flags.cc:384] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0819 06:34:17.406682 2292 flags.cc:384] Enabled unsafe flag: --never_fsync=true W0819 06:34:17.421908 2292 flags.cc:384] Enabled experimental flag: --ipki_ca_key_size=1024 W0819 06:34:17.422055 2292 flags.cc:384] Enabled experimental flag: --ipki_server_key_size=1024 W0819 06:34:17.422188 2292 flags.cc:384] Enabled experimental flag: --tsk_num_rsa_bits=512 I0819 06:34:17.442500 2292 master_main.cc:60] Master server non-default flags: --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/master-0/wal --catalog_manager_wait_for_new_tablets_to_elect_leader=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.126.254:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.4.126.254 --webserver_port=0 --never_fsync=true --heap_profile_path=/tmp/kudu-master.2292 --redact=flag --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.5.0-SNAPSHOT revision ba0f21e2316b78f167bbaba44cab392558d97b6f build type FASTDEBUG built by None at 19 Aug 2017 06:31:02 UTC on jenkins-slave-c3oe build id 9422 TSAN enabled I0819 06:34:17.452407 2292 master_main.cc:67] Initializing master server... I0819 06:34:17.454054 2292 system_ntp.cc:96] NTP initialized. Resolution in nanos?: 1 Skew PPM: 500 Current error: 1302905 I0819 06:34:17.455047 2292 env_posix.cc:1586] Not raising process file limit of 65536; it is already as high as it can go I0819 06:34:17.455356 2292 file_cache.cc:464] Constructed file cache lbm with capacity 26214 I0819 06:34:17.460620 2292 server_base.cc:215] Could not load existing FS layout: Not found: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0819 06:34:17.460758 2292 server_base.cc:216] Creating new FS layout I0819 06:34:17.468431 2292 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data/instance: uuid: "b9cb2c452a0649d48ea63390f3ae4bba" format_stamp: "Formatted at 2017-08-19 06:34:17 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:17.469167 2292 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "b9cb2c452a0649d48ea63390f3ae4bba" format_stamp: "Formatted at 2017-08-19 06:34:17 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:17.479915 2292 fs_manager.cc:280] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s I0819 06:34:17.480058 2292 fs_manager.cc:283] Opened local filesystem: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data,/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/master-0/wal uuid: "b9cb2c452a0649d48ea63390f3ae4bba" format_stamp: "Formatted at 2017-08-19 06:34:17 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:17.480265 2292 fs_report.cc:345] Block manager report -------------------- 1 data directories: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0819 06:34:17.649169 2292 master_main.cc:70] Starting Master server... I0819 06:34:17.650377 2309 process_memory.cc:172] Process hard memory limit is 11.777661 GB I0819 06:34:17.650521 2309 process_memory.cc:174] Process soft memory limit is 9.422129 GB I0819 06:34:17.650615 2309 process_memory.cc:177] Process memory pressure threshold is 7.066597 GB I0819 06:34:17.691237 2292 rpc_server.cc:186] RPC server started. Bound to: 127.4.126.254:58831 I0819 06:34:17.729990 2292 webserver.cc:145] Starting webserver on 127.4.126.254:0 I0819 06:34:17.730103 2292 webserver.cc:156] Document root disabled I0819 06:34:17.731357 2292 webserver.cc:278] Webserver started. Bound to: http://127.4.126.254:51249/ I0819 06:34:17.732147 2292 server_base.cc:359] Dumped server information to /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data/info.pb I0819 06:34:17.734446 1150 external_mini_cluster.cc:814] Started /tmp/run_tha_test479zHR/build/tsan/bin/kudu-master as pid 2292 I0819 06:34:17.735348 1150 external_mini_cluster.cc:752] Running /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data --rpc_bind_addresses=127.4.126.1:0 --local_ip_for_outbound_sockets=127.4.126.1 --webserver_interface=127.4.126.1 --webserver_port=0 --tserver_master_addrs=127.4.126.254:58831 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=flag --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --enable_leader_failure_detection=false with env {} I0819 06:34:17.747957 2353 tablet_bootstrap.cc:402] T 00000000000000000000000000000000 P b9cb2c452a0649d48ea63390f3ae4bba: Bootstrap starting. I0819 06:34:17.750355 2353 tablet_bootstrap.cc:537] T 00000000000000000000000000000000 P b9cb2c452a0649d48ea63390f3ae4bba: No blocks or log segments found. Creating new log. I0819 06:34:17.751562 2353 log.cc:511] T 00000000000000000000000000000000 P b9cb2c452a0649d48ea63390f3ae4bba: Log is configured to *not* fsync() on all Append() calls I0819 06:34:17.754072 2353 tablet_bootstrap.cc:402] T 00000000000000000000000000000000 P b9cb2c452a0649d48ea63390f3ae4bba: No bootstrap required, opened a new log I0819 06:34:17.756563 2353 raft_consensus.cc:286] T 00000000000000000000000000000000 P b9cb2c452a0649d48ea63390f3ae4bba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9cb2c452a0649d48ea63390f3ae4bba" member_type: VOTER } I0819 06:34:17.757055 2353 raft_consensus.cc:318] T 00000000000000000000000000000000 P b9cb2c452a0649d48ea63390f3ae4bba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0819 06:34:17.757212 2353 raft_consensus.cc:574] T 00000000000000000000000000000000 P b9cb2c452a0649d48ea63390f3ae4bba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9cb2c452a0649d48ea63390f3ae4bba, State: Running, Role: FOLLOWER I0819 06:34:17.757426 2353 consensus_queue.cc:190] T 00000000000000000000000000000000 P b9cb2c452a0649d48ea63390f3ae4bba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0819 06:34:17.757655 2353 raft_consensus.cc:329] T 00000000000000000000000000000000 P b9cb2c452a0649d48ea63390f3ae4bba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0819 06:34:17.757812 2353 raft_consensus.cc:413] T 00000000000000000000000000000000 P b9cb2c452a0649d48ea63390f3ae4bba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0819 06:34:17.758002 2353 raft_consensus.cc:2401] T 00000000000000000000000000000000 P b9cb2c452a0649d48ea63390f3ae4bba: Snoozing failure detection for election timeout plus an additional 1.564s I0819 06:34:17.758149 2353 raft_consensus.cc:2448] T 00000000000000000000000000000000 P b9cb2c452a0649d48ea63390f3ae4bba [term 0 FOLLOWER]: Advancing to term 1 I0819 06:34:17.760781 2353 raft_consensus.cc:437] T 00000000000000000000000000000000 P b9cb2c452a0649d48ea63390f3ae4bba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9cb2c452a0649d48ea63390f3ae4bba" member_type: VOTER } I0819 06:34:17.761229 2353 leader_election.cc:244] T 00000000000000000000000000000000 P b9cb2c452a0649d48ea63390f3ae4bba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0819 06:34:17.762862 2357 raft_consensus.cc:2401] T 00000000000000000000000000000000 P b9cb2c452a0649d48ea63390f3ae4bba: Snoozing failure detection for election timeout plus an additional 1.663s I0819 06:34:17.763099 2357 raft_consensus.cc:2205] T 00000000000000000000000000000000 P b9cb2c452a0649d48ea63390f3ae4bba [term 1 FOLLOWER]: Leader election won for term 1 I0819 06:34:17.765091 2357 raft_consensus.cc:540] T 00000000000000000000000000000000 P b9cb2c452a0649d48ea63390f3ae4bba [term 1 LEADER]: Becoming Leader. State: Replica: b9cb2c452a0649d48ea63390f3ae4bba, State: Running, Role: LEADER I0819 06:34:17.765450 2357 consensus_queue.cc:169] T 00000000000000000000000000000000 P b9cb2c452a0649d48ea63390f3ae4bba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9cb2c452a0649d48ea63390f3ae4bba" member_type: VOTER } I0819 06:34:17.784673 2353 sys_catalog.cc:287] T 00000000000000000000000000000000 P b9cb2c452a0649d48ea63390f3ae4bba [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "b9cb2c452a0649d48ea63390f3ae4bba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9cb2c452a0649d48ea63390f3ae4bba" member_type: VOTER } } I0819 06:34:17.785264 2353 sys_catalog.cc:290] T 00000000000000000000000000000000 P b9cb2c452a0649d48ea63390f3ae4bba [sys.catalog]: This master's current role is: LEADER I0819 06:34:17.784580 2359 sys_catalog.cc:287] T 00000000000000000000000000000000 P b9cb2c452a0649d48ea63390f3ae4bba [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9cb2c452a0649d48ea63390f3ae4bba. Latest consensus state: current_term: 1 leader_uuid: "b9cb2c452a0649d48ea63390f3ae4bba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9cb2c452a0649d48ea63390f3ae4bba" member_type: VOTER } } I0819 06:34:17.785821 2359 sys_catalog.cc:290] T 00000000000000000000000000000000 P b9cb2c452a0649d48ea63390f3ae4bba [sys.catalog]: This master's current role is: LEADER I0819 06:34:17.788907 2358 sys_catalog.cc:287] T 00000000000000000000000000000000 P b9cb2c452a0649d48ea63390f3ae4bba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9cb2c452a0649d48ea63390f3ae4bba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9cb2c452a0649d48ea63390f3ae4bba" member_type: VOTER } } I0819 06:34:17.790331 2358 sys_catalog.cc:290] T 00000000000000000000000000000000 P b9cb2c452a0649d48ea63390f3ae4bba [sys.catalog]: This master's current role is: LEADER I0819 06:34:17.790359 2353 sys_catalog.cc:369] T 00000000000000000000000000000000 P b9cb2c452a0649d48ea63390f3ae4bba [sys.catalog]: configured and running, proceeding with master startup. I0819 06:34:17.790993 2361 catalog_manager.cc:1018] Loading table and tablet metadata into memory... I0819 06:34:17.792351 2292 master_main.cc:73] Master server successfully started. I0819 06:34:17.795816 2361 catalog_manager.cc:1029] Initializing Kudu internal certificate authority... I0819 06:34:17.816893 2361 catalog_manager.cc:914] Generated new certificate authority record I0819 06:34:17.819293 2361 catalog_manager.cc:1038] Loading token signing keys... I0819 06:34:17.844264 2361 catalog_manager.cc:3605] T 00000000000000000000000000000000 P b9cb2c452a0649d48ea63390f3ae4bba: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0819 06:34:17.886906 2354 flags.cc:384] Enabled unsafe flag: --enable_leader_failure_detection=false W0819 06:34:17.887336 2354 flags.cc:384] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0819 06:34:17.887545 2354 flags.cc:384] Enabled unsafe flag: --never_fsync=true W0819 06:34:17.901999 2354 flags.cc:384] Enabled experimental flag: --ipki_server_key_size=1024 W0819 06:34:17.902370 2354 flags.cc:384] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.126.1 I0819 06:34:17.922981 2354 tablet_server_main.cc:64] Tablet server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.126.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.4.126.1 --webserver_port=0 --tserver_master_addrs=127.4.126.254:58831 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.2354 --redact=flag --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.126.1 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.5.0-SNAPSHOT revision ba0f21e2316b78f167bbaba44cab392558d97b6f build type FASTDEBUG built by None at 19 Aug 2017 06:31:02 UTC on jenkins-slave-c3oe build id 9422 TSAN enabled I0819 06:34:17.952971 2354 tablet_server_main.cc:71] Initializing tablet server... I0819 06:34:17.954898 2354 system_ntp.cc:96] NTP initialized. Resolution in nanos?: 1 Skew PPM: 500 Current error: 1302905 I0819 06:34:17.955543 2354 env_posix.cc:1586] Not raising process file limit of 65536; it is already as high as it can go I0819 06:34:17.955837 2354 file_cache.cc:464] Constructed file cache lbm with capacity 26214 I0819 06:34:17.960820 2354 server_base.cc:215] Could not load existing FS layout: Not found: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0819 06:34:17.960964 2354 server_base.cc:216] Creating new FS layout I0819 06:34:17.968546 2354 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "2587296a7d144332aeff280fdf54b90f" format_stamp: "Formatted at 2017-08-19 06:34:17 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:17.969326 2354 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "2587296a7d144332aeff280fdf54b90f" format_stamp: "Formatted at 2017-08-19 06:34:17 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:18.001633 2354 fs_manager.cc:280] Time spent opening block manager: real 0.026s user 0.004s sys 0.024s I0819 06:34:18.001787 2354 fs_manager.cc:283] Opened local filesystem: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data,/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/wal uuid: "2587296a7d144332aeff280fdf54b90f" format_stamp: "Formatted at 2017-08-19 06:34:17 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:18.001999 2354 fs_report.cc:345] Block manager report -------------------- 1 data directories: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0819 06:34:18.233247 2354 ts_tablet_manager.cc:190] Loaded tablet metadata (0 live tablets) I0819 06:34:18.244972 2354 tablet_server_main.cc:76] Starting tablet server... I0819 06:34:18.449131 2354 rpc_server.cc:186] RPC server started. Bound to: 127.4.126.1:56226 I0819 06:34:18.450067 2354 webserver.cc:145] Starting webserver on 127.4.126.1:0 I0819 06:34:18.450218 2354 webserver.cc:156] Document root disabled I0819 06:34:18.451653 2354 webserver.cc:278] Webserver started. Bound to: http://127.4.126.1:51735/ I0819 06:34:18.452503 2354 server_base.cc:359] Dumped server information to /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/info.pb I0819 06:34:18.455814 2487 process_memory.cc:172] Process hard memory limit is 11.777661 GB I0819 06:34:18.456647 2487 process_memory.cc:174] Process soft memory limit is 9.422129 GB I0819 06:34:18.456774 2487 process_memory.cc:177] Process memory pressure threshold is 7.066597 GB I0819 06:34:18.457504 1150 external_mini_cluster.cc:814] Started /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver as pid 2354 I0819 06:34:18.458652 1150 external_mini_cluster.cc:752] Running /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data --rpc_bind_addresses=127.4.126.2:0 --local_ip_for_outbound_sockets=127.4.126.2 --webserver_interface=127.4.126.2 --webserver_port=0 --tserver_master_addrs=127.4.126.254:58831 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=flag --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --enable_leader_failure_detection=false with env {} I0819 06:34:18.456521 2354 tablet_server_main.cc:79] Tablet server successfully started. I0819 06:34:18.506695 2486 heartbeater.cc:295] Connected to a master server at 127.4.126.254:58831 I0819 06:34:18.507031 2486 heartbeater.cc:365] Registering TS with master... I0819 06:34:18.507627 2486 heartbeater.cc:392] Master 127.4.126.254:58831 requested a full tablet report, sending... I0819 06:34:18.509402 2320 ts_manager.cc:79] Registered new tserver with Master: 2587296a7d144332aeff280fdf54b90f (127.4.126.1:56226) I0819 06:34:18.510953 2320 master_service.cc:197] Signed X509 certificate for tserver {username='slave'} at 127.4.126.1:52333 WARNING: Logging before InitGoogleLogging() is written to STDERR W0819 06:34:18.661490 2490 flags.cc:384] Enabled unsafe flag: --enable_leader_failure_detection=false W0819 06:34:18.661887 2490 flags.cc:384] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0819 06:34:18.662088 2490 flags.cc:384] Enabled unsafe flag: --never_fsync=true W0819 06:34:18.676558 2490 flags.cc:384] Enabled experimental flag: --ipki_server_key_size=1024 W0819 06:34:18.676905 2490 flags.cc:384] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.126.2 I0819 06:34:18.699056 2490 tablet_server_main.cc:64] Tablet server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.126.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.4.126.2 --webserver_port=0 --tserver_master_addrs=127.4.126.254:58831 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.2490 --redact=flag --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.126.2 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.5.0-SNAPSHOT revision ba0f21e2316b78f167bbaba44cab392558d97b6f build type FASTDEBUG built by None at 19 Aug 2017 06:31:02 UTC on jenkins-slave-c3oe build id 9422 TSAN enabled I0819 06:34:18.728952 2490 tablet_server_main.cc:71] Initializing tablet server... I0819 06:34:18.730821 2490 system_ntp.cc:96] NTP initialized. Resolution in nanos?: 1 Skew PPM: 500 Current error: 1303405 I0819 06:34:18.731462 2490 env_posix.cc:1586] Not raising process file limit of 65536; it is already as high as it can go I0819 06:34:18.731748 2490 file_cache.cc:464] Constructed file cache lbm with capacity 26214 I0819 06:34:18.736709 2490 server_base.cc:215] Could not load existing FS layout: Not found: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0819 06:34:18.736855 2490 server_base.cc:216] Creating new FS layout I0819 06:34:18.744292 2490 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "df90b3557def4c5db6b75f8e6ee27af5" format_stamp: "Formatted at 2017-08-19 06:34:18 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:18.745031 2490 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "df90b3557def4c5db6b75f8e6ee27af5" format_stamp: "Formatted at 2017-08-19 06:34:18 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:18.777571 2490 fs_manager.cc:280] Time spent opening block manager: real 0.025s user 0.008s sys 0.020s I0819 06:34:18.777727 2490 fs_manager.cc:283] Opened local filesystem: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data,/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/wal uuid: "df90b3557def4c5db6b75f8e6ee27af5" format_stamp: "Formatted at 2017-08-19 06:34:18 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:18.777943 2490 fs_report.cc:345] Block manager report -------------------- 1 data directories: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0819 06:34:19.038483 2490 ts_tablet_manager.cc:190] Loaded tablet metadata (0 live tablets) I0819 06:34:19.039827 2490 tablet_server_main.cc:76] Starting tablet server... I0819 06:34:19.134654 2490 rpc_server.cc:186] RPC server started. Bound to: 127.4.126.2:35352 I0819 06:34:19.135538 2490 webserver.cc:145] Starting webserver on 127.4.126.2:0 I0819 06:34:19.135640 2490 webserver.cc:156] Document root disabled I0819 06:34:19.136863 2490 webserver.cc:278] Webserver started. Bound to: http://127.4.126.2:48305/ I0819 06:34:19.137573 2490 server_base.cc:359] Dumped server information to /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/info.pb I0819 06:34:19.139351 1150 external_mini_cluster.cc:814] Started /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver as pid 2490 I0819 06:34:19.140326 1150 external_mini_cluster.cc:752] Running /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data --rpc_bind_addresses=127.4.126.3:0 --local_ip_for_outbound_sockets=127.4.126.3 --webserver_interface=127.4.126.3 --webserver_port=0 --tserver_master_addrs=127.4.126.254:58831 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=flag --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --enable_leader_failure_detection=false with env {} I0819 06:34:19.153780 2490 tablet_server_main.cc:79] Tablet server successfully started. I0819 06:34:19.172271 2611 heartbeater.cc:295] Connected to a master server at 127.4.126.254:58831 I0819 06:34:19.172626 2611 heartbeater.cc:365] Registering TS with master... I0819 06:34:19.173162 2611 heartbeater.cc:392] Master 127.4.126.254:58831 requested a full tablet report, sending... I0819 06:34:19.175019 2320 ts_manager.cc:79] Registered new tserver with Master: df90b3557def4c5db6b75f8e6ee27af5 (127.4.126.2:35352) I0819 06:34:19.176666 2320 master_service.cc:197] Signed X509 certificate for tserver {username='slave'} at 127.4.126.2:52920 I0819 06:34:19.154250 2612 process_memory.cc:172] Process hard memory limit is 11.777661 GB I0819 06:34:19.180662 2612 process_memory.cc:174] Process soft memory limit is 9.422129 GB I0819 06:34:19.180757 2612 process_memory.cc:177] Process memory pressure threshold is 7.066597 GB WARNING: Logging before InitGoogleLogging() is written to STDERR W0819 06:34:19.272526 2615 flags.cc:384] Enabled unsafe flag: --enable_leader_failure_detection=false W0819 06:34:19.272900 2615 flags.cc:384] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0819 06:34:19.273092 2615 flags.cc:384] Enabled unsafe flag: --never_fsync=true W0819 06:34:19.287437 2615 flags.cc:384] Enabled experimental flag: --ipki_server_key_size=1024 W0819 06:34:19.287781 2615 flags.cc:384] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.126.3 I0819 06:34:19.307174 2615 tablet_server_main.cc:64] Tablet server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.126.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.4.126.3 --webserver_port=0 --tserver_master_addrs=127.4.126.254:58831 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.2615 --redact=flag --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.126.3 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.5.0-SNAPSHOT revision ba0f21e2316b78f167bbaba44cab392558d97b6f build type FASTDEBUG built by None at 19 Aug 2017 06:31:02 UTC on jenkins-slave-c3oe build id 9422 TSAN enabled I0819 06:34:19.317493 2615 tablet_server_main.cc:71] Initializing tablet server... I0819 06:34:19.319263 2615 system_ntp.cc:96] NTP initialized. Resolution in nanos?: 1 Skew PPM: 500 Current error: 1303905 I0819 06:34:19.319850 2615 env_posix.cc:1586] Not raising process file limit of 65536; it is already as high as it can go I0819 06:34:19.320125 2615 file_cache.cc:464] Constructed file cache lbm with capacity 26214 I0819 06:34:19.325107 2615 server_base.cc:215] Could not load existing FS layout: Not found: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0819 06:34:19.325243 2615 server_base.cc:216] Creating new FS layout I0819 06:34:19.332767 2615 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "d870d0b156694601ace827bffc47e332" format_stamp: "Formatted at 2017-08-19 06:34:19 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:19.333528 2615 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "d870d0b156694601ace827bffc47e332" format_stamp: "Formatted at 2017-08-19 06:34:19 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:19.344204 2615 fs_manager.cc:280] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s I0819 06:34:19.344347 2615 fs_manager.cc:283] Opened local filesystem: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data,/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/wal uuid: "d870d0b156694601ace827bffc47e332" format_stamp: "Formatted at 2017-08-19 06:34:19 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:19.344554 2615 fs_report.cc:345] Block manager report -------------------- 1 data directories: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0819 06:34:19.514161 2486 heartbeater.cc:384] Master 127.4.126.254:58831 was elected leader, sending a full tablet report... I0819 06:34:19.537271 2615 ts_tablet_manager.cc:190] Loaded tablet metadata (0 live tablets) I0819 06:34:19.538395 2615 tablet_server_main.cc:76] Starting tablet server... I0819 06:34:19.629261 2615 rpc_server.cc:186] RPC server started. Bound to: 127.4.126.3:40253 I0819 06:34:19.630139 2615 webserver.cc:145] Starting webserver on 127.4.126.3:0 I0819 06:34:19.630240 2615 webserver.cc:156] Document root disabled I0819 06:34:19.631441 2615 webserver.cc:278] Webserver started. Bound to: http://127.4.126.3:40073/ I0819 06:34:19.632179 2615 server_base.cc:359] Dumped server information to /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestLeaderStepDown.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/info.pb I0819 06:34:19.634652 1150 external_mini_cluster.cc:814] Started /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver as pid 2615 I0819 06:34:19.635177 2615 tablet_server_main.cc:79] Tablet server successfully started. I0819 06:34:19.658737 2736 heartbeater.cc:295] Connected to a master server at 127.4.126.254:58831 I0819 06:34:19.659029 2736 heartbeater.cc:365] Registering TS with master... I0819 06:34:19.659826 2736 heartbeater.cc:392] Master 127.4.126.254:58831 requested a full tablet report, sending... I0819 06:34:19.661532 2320 ts_manager.cc:79] Registered new tserver with Master: d870d0b156694601ace827bffc47e332 (127.4.126.3:40253) I0819 06:34:19.662587 2320 master_service.cc:197] Signed X509 certificate for tserver {username='slave'} at 127.4.126.3:55138 I0819 06:34:19.637061 2737 process_memory.cc:172] Process hard memory limit is 11.777661 GB I0819 06:34:19.677166 2737 process_memory.cc:174] Process soft memory limit is 9.422129 GB I0819 06:34:19.677284 2737 process_memory.cc:177] Process memory pressure threshold is 7.066597 GB I0819 06:34:19.680553 1150 external_mini_cluster.cc:454] 3 TS(s) registered with all masters I0819 06:34:19.700472 2320 catalog_manager.cc:1281] CreateTable from {username='slave'} at 127.0.0.1:37302: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0819 06:34:19.749433 2423 tablet_service.cc:684] Processing CreateTablet for tablet 60b28a975fb04ca2a1c0c8151ee3765b (table=TestTable [id=349b61d778e84ed9be15c83727173c9d]), partition=RANGE (key) PARTITION UNBOUNDED I0819 06:34:19.750133 2548 tablet_service.cc:684] Processing CreateTablet for tablet 60b28a975fb04ca2a1c0c8151ee3765b (table=TestTable [id=349b61d778e84ed9be15c83727173c9d]), partition=RANGE (key) PARTITION UNBOUNDED I0819 06:34:19.757385 2673 tablet_service.cc:684] Processing CreateTablet for tablet 60b28a975fb04ca2a1c0c8151ee3765b (table=TestTable [id=349b61d778e84ed9be15c83727173c9d]), partition=RANGE (key) PARTITION UNBOUNDED I0819 06:34:19.760624 2423 ts_tablet_manager.cc:877] T 60b28a975fb04ca2a1c0c8151ee3765b P 2587296a7d144332aeff280fdf54b90f: Registered tablet (data state: TABLET_DATA_READY) I0819 06:34:19.763537 2756 ts_tablet_manager.cc:740] T 60b28a975fb04ca2a1c0c8151ee3765b P 2587296a7d144332aeff280fdf54b90f: Bootstrapping tablet I0819 06:34:19.764077 2756 tablet_bootstrap.cc:402] T 60b28a975fb04ca2a1c0c8151ee3765b P 2587296a7d144332aeff280fdf54b90f: Bootstrap starting. I0819 06:34:19.765714 2548 ts_tablet_manager.cc:877] T 60b28a975fb04ca2a1c0c8151ee3765b P df90b3557def4c5db6b75f8e6ee27af5: Registered tablet (data state: TABLET_DATA_READY) I0819 06:34:19.767822 2756 tablet_bootstrap.cc:537] T 60b28a975fb04ca2a1c0c8151ee3765b P 2587296a7d144332aeff280fdf54b90f: No blocks or log segments found. Creating new log. I0819 06:34:19.769546 2756 log.cc:511] T 60b28a975fb04ca2a1c0c8151ee3765b P 2587296a7d144332aeff280fdf54b90f: Log is configured to *not* fsync() on all Append() calls I0819 06:34:19.773190 2757 ts_tablet_manager.cc:740] T 60b28a975fb04ca2a1c0c8151ee3765b P df90b3557def4c5db6b75f8e6ee27af5: Bootstrapping tablet I0819 06:34:19.774081 2757 tablet_bootstrap.cc:402] T 60b28a975fb04ca2a1c0c8151ee3765b P df90b3557def4c5db6b75f8e6ee27af5: Bootstrap starting. I0819 06:34:19.775192 2673 ts_tablet_manager.cc:877] T 60b28a975fb04ca2a1c0c8151ee3765b P d870d0b156694601ace827bffc47e332: Registered tablet (data state: TABLET_DATA_READY) I0819 06:34:19.778139 2759 ts_tablet_manager.cc:740] T 60b28a975fb04ca2a1c0c8151ee3765b P d870d0b156694601ace827bffc47e332: Bootstrapping tablet I0819 06:34:19.778437 2757 tablet_bootstrap.cc:537] T 60b28a975fb04ca2a1c0c8151ee3765b P df90b3557def4c5db6b75f8e6ee27af5: No blocks or log segments found. Creating new log. I0819 06:34:19.778606 2759 tablet_bootstrap.cc:402] T 60b28a975fb04ca2a1c0c8151ee3765b P d870d0b156694601ace827bffc47e332: Bootstrap starting. I0819 06:34:19.780144 2757 log.cc:511] T 60b28a975fb04ca2a1c0c8151ee3765b P df90b3557def4c5db6b75f8e6ee27af5: Log is configured to *not* fsync() on all Append() calls I0819 06:34:19.782030 2756 tablet_bootstrap.cc:402] T 60b28a975fb04ca2a1c0c8151ee3765b P 2587296a7d144332aeff280fdf54b90f: No bootstrap required, opened a new log I0819 06:34:19.782389 2759 tablet_bootstrap.cc:537] T 60b28a975fb04ca2a1c0c8151ee3765b P d870d0b156694601ace827bffc47e332: No blocks or log segments found. Creating new log. I0819 06:34:19.782479 2756 ts_tablet_manager.cc:752] T 60b28a975fb04ca2a1c0c8151ee3765b P 2587296a7d144332aeff280fdf54b90f: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.004s I0819 06:34:19.783149 2757 tablet_bootstrap.cc:402] T 60b28a975fb04ca2a1c0c8151ee3765b P df90b3557def4c5db6b75f8e6ee27af5: No bootstrap required, opened a new log I0819 06:34:19.783517 2757 ts_tablet_manager.cc:752] T 60b28a975fb04ca2a1c0c8151ee3765b P df90b3557def4c5db6b75f8e6ee27af5: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s I0819 06:34:19.783787 2759 log.cc:511] T 60b28a975fb04ca2a1c0c8151ee3765b P d870d0b156694601ace827bffc47e332: Log is configured to *not* fsync() on all Append() calls I0819 06:34:19.784436 2756 raft_consensus.cc:286] T 60b28a975fb04ca2a1c0c8151ee3765b P 2587296a7d144332aeff280fdf54b90f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df90b3557def4c5db6b75f8e6ee27af5" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 35352 } } peers { permanent_uuid: "2587296a7d144332aeff280fdf54b90f" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 56226 } } peers { permanent_uuid: "d870d0b156694601ace827bffc47e332" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 40253 } } I0819 06:34:19.785358 2756 raft_consensus.cc:574] T 60b28a975fb04ca2a1c0c8151ee3765b P 2587296a7d144332aeff280fdf54b90f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2587296a7d144332aeff280fdf54b90f, State: Running, Role: FOLLOWER I0819 06:34:19.785670 2756 consensus_queue.cc:190] T 60b28a975fb04ca2a1c0c8151ee3765b P 2587296a7d144332aeff280fdf54b90f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0819 06:34:19.786587 2759 tablet_bootstrap.cc:402] T 60b28a975fb04ca2a1c0c8151ee3765b P d870d0b156694601ace827bffc47e332: No bootstrap required, opened a new log I0819 06:34:19.786973 2759 ts_tablet_manager.cc:752] T 60b28a975fb04ca2a1c0c8151ee3765b P d870d0b156694601ace827bffc47e332: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.004s I0819 06:34:19.788172 2756 ts_tablet_manager.cc:781] T 60b28a975fb04ca2a1c0c8151ee3765b P 2587296a7d144332aeff280fdf54b90f: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s I0819 06:34:19.788583 2759 raft_consensus.cc:286] T 60b28a975fb04ca2a1c0c8151ee3765b P d870d0b156694601ace827bffc47e332 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df90b3557def4c5db6b75f8e6ee27af5" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 35352 } } peers { permanent_uuid: "2587296a7d144332aeff280fdf54b90f" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 56226 } } peers { permanent_uuid: "d870d0b156694601ace827bffc47e332" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 40253 } } I0819 06:34:19.789212 2759 raft_consensus.cc:574] T 60b28a975fb04ca2a1c0c8151ee3765b P d870d0b156694601ace827bffc47e332 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870d0b156694601ace827bffc47e332, State: Running, Role: FOLLOWER I0819 06:34:19.789496 2759 consensus_queue.cc:190] T 60b28a975fb04ca2a1c0c8151ee3765b P d870d0b156694601ace827bffc47e332 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0819 06:34:19.788806 2757 raft_consensus.cc:286] T 60b28a975fb04ca2a1c0c8151ee3765b P df90b3557def4c5db6b75f8e6ee27af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df90b3557def4c5db6b75f8e6ee27af5" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 35352 } } peers { permanent_uuid: "2587296a7d144332aeff280fdf54b90f" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 56226 } } peers { permanent_uuid: "d870d0b156694601ace827bffc47e332" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 40253 } } I0819 06:34:19.791193 2757 raft_consensus.cc:574] T 60b28a975fb04ca2a1c0c8151ee3765b P df90b3557def4c5db6b75f8e6ee27af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df90b3557def4c5db6b75f8e6ee27af5, State: Running, Role: FOLLOWER I0819 06:34:19.791492 2757 consensus_queue.cc:190] T 60b28a975fb04ca2a1c0c8151ee3765b P df90b3557def4c5db6b75f8e6ee27af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0819 06:34:19.794713 2611 heartbeater.cc:384] Master 127.4.126.254:58831 was elected leader, sending a full tablet report... I0819 06:34:19.802048 2757 ts_tablet_manager.cc:781] T 60b28a975fb04ca2a1c0c8151ee3765b P df90b3557def4c5db6b75f8e6ee27af5: Time spent starting tablet: real 0.018s user 0.008s sys 0.000s I0819 06:34:19.803573 2736 heartbeater.cc:384] Master 127.4.126.254:58831 was elected leader, sending a full tablet report... I0819 06:34:19.804000 2759 ts_tablet_manager.cc:781] T 60b28a975fb04ca2a1c0c8151ee3765b P d870d0b156694601ace827bffc47e332: Time spent starting tablet: real 0.017s user 0.004s sys 0.008s I0819 06:34:19.805227 1150 external_mini_cluster.cc:454] 3 TS(s) registered with all masters I0819 06:34:19.807314 1150 ts_itest-base.h:198] Waiting for 1 tablets on tserver 2587296a7d144332aeff280fdf54b90f to finish bootstrapping I0819 06:34:19.817879 1150 ts_itest-base.h:198] Waiting for 1 tablets on tserver df90b3557def4c5db6b75f8e6ee27af5 to finish bootstrapping I0819 06:34:19.827038 1150 ts_itest-base.h:198] Waiting for 1 tablets on tserver d870d0b156694601ace827bffc47e332 to finish bootstrapping I0819 06:34:19.846720 2443 raft_consensus.cc:413] T 60b28a975fb04ca2a1c0c8151ee3765b P 2587296a7d144332aeff280fdf54b90f [term 0 FOLLOWER]: Starting forced leader election (received explicit request) I0819 06:34:19.846930 2443 raft_consensus.cc:2448] T 60b28a975fb04ca2a1c0c8151ee3765b P 2587296a7d144332aeff280fdf54b90f [term 0 FOLLOWER]: Advancing to term 1 I0819 06:34:19.849731 2443 raft_consensus.cc:437] T 60b28a975fb04ca2a1c0c8151ee3765b P 2587296a7d144332aeff280fdf54b90f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df90b3557def4c5db6b75f8e6ee27af5" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 35352 } } peers { permanent_uuid: "2587296a7d144332aeff280fdf54b90f" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 56226 } } peers { permanent_uuid: "d870d0b156694601ace827bffc47e332" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 40253 } } I0819 06:34:19.850739 2443 leader_election.cc:217] T 60b28a975fb04ca2a1c0c8151ee3765b P 2587296a7d144332aeff280fdf54b90f [CANDIDATE]: Term 1 election: Requesting vote from peer df90b3557def4c5db6b75f8e6ee27af5 I0819 06:34:19.851184 2443 leader_election.cc:217] T 60b28a975fb04ca2a1c0c8151ee3765b P 2587296a7d144332aeff280fdf54b90f [CANDIDATE]: Term 1 election: Requesting vote from peer d870d0b156694601ace827bffc47e332 I0819 06:34:19.866642 2693 raft_consensus.cc:2448] T 60b28a975fb04ca2a1c0c8151ee3765b P d870d0b156694601ace827bffc47e332 [term 0 FOLLOWER]: Advancing to term 1 I0819 06:34:19.867091 2568 raft_consensus.cc:2448] T 60b28a975fb04ca2a1c0c8151ee3765b P df90b3557def4c5db6b75f8e6ee27af5 [term 0 FOLLOWER]: Advancing to term 1 I0819 06:34:19.869753 2693 raft_consensus.cc:1967] T 60b28a975fb04ca2a1c0c8151ee3765b P d870d0b156694601ace827bffc47e332 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2587296a7d144332aeff280fdf54b90f in term 1. I0819 06:34:19.869886 2568 raft_consensus.cc:1967] T 60b28a975fb04ca2a1c0c8151ee3765b P df90b3557def4c5db6b75f8e6ee27af5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2587296a7d144332aeff280fdf54b90f in term 1. I0819 06:34:19.870739 2374 leader_election.cc:349] T 60b28a975fb04ca2a1c0c8151ee3765b P 2587296a7d144332aeff280fdf54b90f [CANDIDATE]: Term 1 election: Vote granted by peer d870d0b156694601ace827bffc47e332 I0819 06:34:19.871032 2376 leader_election.cc:349] T 60b28a975fb04ca2a1c0c8151ee3765b P 2587296a7d144332aeff280fdf54b90f [CANDIDATE]: Term 1 election: Vote granted by peer df90b3557def4c5db6b75f8e6ee27af5 I0819 06:34:19.871217 2376 leader_election.cc:244] T 60b28a975fb04ca2a1c0c8151ee3765b P 2587296a7d144332aeff280fdf54b90f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0819 06:34:19.871758 2762 raft_consensus.cc:2205] T 60b28a975fb04ca2a1c0c8151ee3765b P 2587296a7d144332aeff280fdf54b90f [term 1 FOLLOWER]: Leader election won for term 1 I0819 06:34:19.873165 2762 raft_consensus.cc:540] T 60b28a975fb04ca2a1c0c8151ee3765b P 2587296a7d144332aeff280fdf54b90f [term 1 LEADER]: Becoming Leader. State: Replica: 2587296a7d144332aeff280fdf54b90f, State: Running, Role: LEADER I0819 06:34:19.873497 2762 consensus_queue.cc:169] T 60b28a975fb04ca2a1c0c8151ee3765b P 2587296a7d144332aeff280fdf54b90f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df90b3557def4c5db6b75f8e6ee27af5" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 35352 } } peers { permanent_uuid: "2587296a7d144332aeff280fdf54b90f" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 56226 } } peers { permanent_uuid: "d870d0b156694601ace827bffc47e332" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 40253 } } I0819 06:34:19.887326 2319 catalog_manager.cc:2559] T 60b28a975fb04ca2a1c0c8151ee3765b reported consensus state change: term changed from 0 to 1, leader changed from to 2587296a7d144332aeff280fdf54b90f (127.4.126.1). New consensus state: current_term: 1 leader_uuid: "2587296a7d144332aeff280fdf54b90f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df90b3557def4c5db6b75f8e6ee27af5" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 35352 } } peers { permanent_uuid: "2587296a7d144332aeff280fdf54b90f" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 56226 } } peers { permanent_uuid: "d870d0b156694601ace827bffc47e332" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 40253 } } } I0819 06:34:19.896715 2693 raft_consensus.cc:961] T 60b28a975fb04ca2a1c0c8151ee3765b P d870d0b156694601ace827bffc47e332 [term 1 FOLLOWER]: Refusing update from remote peer 2587296a7d144332aeff280fdf54b90f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0819 06:34:19.896715 2568 raft_consensus.cc:961] T 60b28a975fb04ca2a1c0c8151ee3765b P df90b3557def4c5db6b75f8e6ee27af5 [term 1 FOLLOWER]: Refusing update from remote peer 2587296a7d144332aeff280fdf54b90f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0819 06:34:19.898002 2768 consensus_queue.cc:726] T 60b28a975fb04ca2a1c0c8151ee3765b P 2587296a7d144332aeff280fdf54b90f [LEADER]: Connected to new peer: Peer: df90b3557def4c5db6b75f8e6ee27af5, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I0819 06:34:19.898305 2762 consensus_queue.cc:726] T 60b28a975fb04ca2a1c0c8151ee3765b P 2587296a7d144332aeff280fdf54b90f [LEADER]: Connected to new peer: Peer: d870d0b156694601ace827bffc47e332, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I0819 06:34:19.949053 2443 raft_consensus.cc:574] T 60b28a975fb04ca2a1c0c8151ee3765b P 2587296a7d144332aeff280fdf54b90f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2587296a7d144332aeff280fdf54b90f, State: Running, Role: LEADER I0819 06:34:19.949350 2443 consensus_queue.cc:190] T 60b28a975fb04ca2a1c0c8151ee3765b P 2587296a7d144332aeff280fdf54b90f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 2, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 1, Mode: NON_LEADER I0819 06:34:19.949530 2443 consensus_peers.cc:418] T 60b28a975fb04ca2a1c0c8151ee3765b P 2587296a7d144332aeff280fdf54b90f -> Peer d870d0b156694601ace827bffc47e332 (127.4.126.3:40253): Closing peer: d870d0b156694601ace827bffc47e332 I0819 06:34:19.949679 2443 consensus_peers.cc:418] T 60b28a975fb04ca2a1c0c8151ee3765b P 2587296a7d144332aeff280fdf54b90f -> Peer df90b3557def4c5db6b75f8e6ee27af5 (127.4.126.2:35352): Closing peer: df90b3557def4c5db6b75f8e6ee27af5 I0819 06:34:19.955108 1150 external_mini_cluster.cc:944] Killing /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver with pid 2354 W0819 06:34:19.975978 2305 connection.cc:551] client connection to 127.4.126.1:56226 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:19.976017 2291 connection.cc:551] client connection to 127.4.126.1:56226 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:19.976932 2290 connection.cc:551] client connection to 127.4.126.1:56226 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:19.976966 2303 connection.cc:551] server connection from 127.4.126.1:52333 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:19.977906 2503 connection.cc:551] server connection from 127.4.126.1:56132 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0819 06:34:19.978946 1150 external_mini_cluster.cc:944] Killing /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver with pid 2490 W0819 06:34:19.978888 2626 connection.cc:551] server connection from 127.4.126.1:35743 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:19.997531 2303 connection.cc:551] client connection to 127.4.126.2:35352 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:19.997581 2291 connection.cc:551] client connection to 127.4.126.2:35352 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:19.998374 2301 connection.cc:551] server connection from 127.4.126.2:52920 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:19.998450 2289 connection.cc:551] client connection to 127.4.126.2:35352 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0819 06:34:19.999367 1150 external_mini_cluster.cc:944] Killing /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver with pid 2615 W0819 06:34:20.017334 2301 connection.cc:551] client connection to 127.4.126.3:40253 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0819 06:34:20.017616 1150 external_mini_cluster.cc:944] Killing /tmp/run_tha_test479zHR/build/tsan/bin/kudu-master with pid 2292 W0819 06:34:20.018087 2287 connection.cc:551] client connection to 127.4.126.3:40253 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:20.020702 2291 connection.cc:551] client connection to 127.4.126.3:40253 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:20.041790 2291 connection.cc:551] client connection to 127.4.126.254:58831 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) [ OK ] RaftConsensusITest.TestLeaderStepDown (2785 ms) [ RUN ] RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure I0819 06:34:20.059366 1150 test_util.cc:185] Using random seed: 1870576332 I0819 06:34:20.062479 1150 ts_itest-base.h:81] Starting cluster with: I0819 06:34:20.062585 1150 ts_itest-base.h:82] -------------- I0819 06:34:20.062661 1150 ts_itest-base.h:83] 3 tablet servers I0819 06:34:20.062739 1150 ts_itest-base.h:84] 3 replicas per TS I0819 06:34:20.062813 1150 ts_itest-base.h:85] -------------- I0819 06:34:20.065373 1150 external_mini_cluster.cc:752] Running /tmp/run_tha_test479zHR/build/tsan/bin/kudu-master /tmp/run_tha_test479zHR/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.126.254:0 --webserver_interface=127.4.126.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=flag --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0819 06:34:20.192050 2785 flags.cc:384] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0819 06:34:20.192442 2785 flags.cc:384] Enabled unsafe flag: --never_fsync=true W0819 06:34:20.208030 2785 flags.cc:384] Enabled experimental flag: --ipki_ca_key_size=1024 W0819 06:34:20.208178 2785 flags.cc:384] Enabled experimental flag: --ipki_server_key_size=1024 W0819 06:34:20.208276 2785 flags.cc:384] Enabled experimental flag: --tsk_num_rsa_bits=512 I0819 06:34:20.229120 2785 master_main.cc:60] Master server non-default flags: --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/master-0/wal --catalog_manager_wait_for_new_tablets_to_elect_leader=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.126.254:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.4.126.254 --webserver_port=0 --never_fsync=true --heap_profile_path=/tmp/kudu-master.2785 --redact=flag --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.5.0-SNAPSHOT revision ba0f21e2316b78f167bbaba44cab392558d97b6f build type FASTDEBUG built by None at 19 Aug 2017 06:31:02 UTC on jenkins-slave-c3oe build id 9422 TSAN enabled I0819 06:34:20.247714 2785 master_main.cc:67] Initializing master server... I0819 06:34:20.249387 2785 system_ntp.cc:96] NTP initialized. Resolution in nanos?: 1 Skew PPM: 500 Current error: 1304405 I0819 06:34:20.250340 2785 env_posix.cc:1586] Not raising process file limit of 65536; it is already as high as it can go I0819 06:34:20.250623 2785 file_cache.cc:464] Constructed file cache lbm with capacity 26214 I0819 06:34:20.255635 2785 server_base.cc:215] Could not load existing FS layout: Not found: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0819 06:34:20.255790 2785 server_base.cc:216] Creating new FS layout I0819 06:34:20.263267 2785 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data/instance: uuid: "280edc58eb9248bf99bb8e42b4eae2b6" format_stamp: "Formatted at 2017-08-19 06:34:20 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:20.263988 2785 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "280edc58eb9248bf99bb8e42b4eae2b6" format_stamp: "Formatted at 2017-08-19 06:34:20 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:20.274546 2785 fs_manager.cc:280] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s I0819 06:34:20.274686 2785 fs_manager.cc:283] Opened local filesystem: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data,/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/master-0/wal uuid: "280edc58eb9248bf99bb8e42b4eae2b6" format_stamp: "Formatted at 2017-08-19 06:34:20 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:20.274870 2785 fs_report.cc:345] Block manager report -------------------- 1 data directories: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0819 06:34:20.449306 2785 master_main.cc:70] Starting Master server... I0819 06:34:20.450523 2802 process_memory.cc:172] Process hard memory limit is 11.777661 GB I0819 06:34:20.450681 2802 process_memory.cc:174] Process soft memory limit is 9.422129 GB I0819 06:34:20.450778 2802 process_memory.cc:177] Process memory pressure threshold is 7.066597 GB I0819 06:34:20.491711 2785 rpc_server.cc:186] RPC server started. Bound to: 127.4.126.254:39962 I0819 06:34:20.530879 2785 webserver.cc:145] Starting webserver on 127.4.126.254:0 I0819 06:34:20.531013 2785 webserver.cc:156] Document root disabled I0819 06:34:20.532263 2785 webserver.cc:278] Webserver started. Bound to: http://127.4.126.254:36180/ I0819 06:34:20.533068 2785 server_base.cc:359] Dumped server information to /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data/info.pb I0819 06:34:20.541280 1150 external_mini_cluster.cc:814] Started /tmp/run_tha_test479zHR/build/tsan/bin/kudu-master as pid 2785 I0819 06:34:20.542268 1150 external_mini_cluster.cc:752] Running /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data --rpc_bind_addresses=127.4.126.1:0 --local_ip_for_outbound_sockets=127.4.126.1 --webserver_interface=127.4.126.1 --webserver_port=0 --tserver_master_addrs=127.4.126.254:39962 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=flag --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --memory_limit_hard_bytes=4194304 --enable_leader_failure_detection=false --tablet_transaction_memory_limit_mb=-1 with env {} I0819 06:34:20.548866 2846 tablet_bootstrap.cc:402] T 00000000000000000000000000000000 P 280edc58eb9248bf99bb8e42b4eae2b6: Bootstrap starting. I0819 06:34:20.551403 2846 tablet_bootstrap.cc:537] T 00000000000000000000000000000000 P 280edc58eb9248bf99bb8e42b4eae2b6: No blocks or log segments found. Creating new log. I0819 06:34:20.552736 2846 log.cc:511] T 00000000000000000000000000000000 P 280edc58eb9248bf99bb8e42b4eae2b6: Log is configured to *not* fsync() on all Append() calls I0819 06:34:20.555414 2846 tablet_bootstrap.cc:402] T 00000000000000000000000000000000 P 280edc58eb9248bf99bb8e42b4eae2b6: No bootstrap required, opened a new log I0819 06:34:20.557991 2846 raft_consensus.cc:286] T 00000000000000000000000000000000 P 280edc58eb9248bf99bb8e42b4eae2b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280edc58eb9248bf99bb8e42b4eae2b6" member_type: VOTER } I0819 06:34:20.558488 2846 raft_consensus.cc:318] T 00000000000000000000000000000000 P 280edc58eb9248bf99bb8e42b4eae2b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0819 06:34:20.558636 2846 raft_consensus.cc:574] T 00000000000000000000000000000000 P 280edc58eb9248bf99bb8e42b4eae2b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 280edc58eb9248bf99bb8e42b4eae2b6, State: Running, Role: FOLLOWER I0819 06:34:20.558848 2846 consensus_queue.cc:190] T 00000000000000000000000000000000 P 280edc58eb9248bf99bb8e42b4eae2b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0819 06:34:20.559109 2846 raft_consensus.cc:329] T 00000000000000000000000000000000 P 280edc58eb9248bf99bb8e42b4eae2b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0819 06:34:20.559272 2846 raft_consensus.cc:413] T 00000000000000000000000000000000 P 280edc58eb9248bf99bb8e42b4eae2b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0819 06:34:20.559469 2846 raft_consensus.cc:2401] T 00000000000000000000000000000000 P 280edc58eb9248bf99bb8e42b4eae2b6: Snoozing failure detection for election timeout plus an additional 1.684s I0819 06:34:20.559625 2846 raft_consensus.cc:2448] T 00000000000000000000000000000000 P 280edc58eb9248bf99bb8e42b4eae2b6 [term 0 FOLLOWER]: Advancing to term 1 I0819 06:34:20.562348 2846 raft_consensus.cc:437] T 00000000000000000000000000000000 P 280edc58eb9248bf99bb8e42b4eae2b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280edc58eb9248bf99bb8e42b4eae2b6" member_type: VOTER } I0819 06:34:20.562819 2846 leader_election.cc:244] T 00000000000000000000000000000000 P 280edc58eb9248bf99bb8e42b4eae2b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0819 06:34:20.564481 2850 raft_consensus.cc:2401] T 00000000000000000000000000000000 P 280edc58eb9248bf99bb8e42b4eae2b6: Snoozing failure detection for election timeout plus an additional 2.097s I0819 06:34:20.564725 2850 raft_consensus.cc:2205] T 00000000000000000000000000000000 P 280edc58eb9248bf99bb8e42b4eae2b6 [term 1 FOLLOWER]: Leader election won for term 1 I0819 06:34:20.566720 2850 raft_consensus.cc:540] T 00000000000000000000000000000000 P 280edc58eb9248bf99bb8e42b4eae2b6 [term 1 LEADER]: Becoming Leader. State: Replica: 280edc58eb9248bf99bb8e42b4eae2b6, State: Running, Role: LEADER I0819 06:34:20.567085 2850 consensus_queue.cc:169] T 00000000000000000000000000000000 P 280edc58eb9248bf99bb8e42b4eae2b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280edc58eb9248bf99bb8e42b4eae2b6" member_type: VOTER } I0819 06:34:20.586067 2846 sys_catalog.cc:287] T 00000000000000000000000000000000 P 280edc58eb9248bf99bb8e42b4eae2b6 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "280edc58eb9248bf99bb8e42b4eae2b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280edc58eb9248bf99bb8e42b4eae2b6" member_type: VOTER } } I0819 06:34:20.586557 2846 sys_catalog.cc:290] T 00000000000000000000000000000000 P 280edc58eb9248bf99bb8e42b4eae2b6 [sys.catalog]: This master's current role is: LEADER I0819 06:34:20.587144 2851 sys_catalog.cc:287] T 00000000000000000000000000000000 P 280edc58eb9248bf99bb8e42b4eae2b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "280edc58eb9248bf99bb8e42b4eae2b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280edc58eb9248bf99bb8e42b4eae2b6" member_type: VOTER } } I0819 06:34:20.587927 2851 sys_catalog.cc:290] T 00000000000000000000000000000000 P 280edc58eb9248bf99bb8e42b4eae2b6 [sys.catalog]: This master's current role is: LEADER I0819 06:34:20.588965 2852 sys_catalog.cc:287] T 00000000000000000000000000000000 P 280edc58eb9248bf99bb8e42b4eae2b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 280edc58eb9248bf99bb8e42b4eae2b6. Latest consensus state: current_term: 1 leader_uuid: "280edc58eb9248bf99bb8e42b4eae2b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280edc58eb9248bf99bb8e42b4eae2b6" member_type: VOTER } } I0819 06:34:20.589373 2852 sys_catalog.cc:290] T 00000000000000000000000000000000 P 280edc58eb9248bf99bb8e42b4eae2b6 [sys.catalog]: This master's current role is: LEADER I0819 06:34:20.590950 2846 sys_catalog.cc:369] T 00000000000000000000000000000000 P 280edc58eb9248bf99bb8e42b4eae2b6 [sys.catalog]: configured and running, proceeding with master startup. I0819 06:34:20.591248 2854 catalog_manager.cc:1018] Loading table and tablet metadata into memory... I0819 06:34:20.592833 2785 master_main.cc:73] Master server successfully started. I0819 06:34:20.596346 2854 catalog_manager.cc:1029] Initializing Kudu internal certificate authority... I0819 06:34:20.650072 2854 catalog_manager.cc:914] Generated new certificate authority record I0819 06:34:20.652637 2854 catalog_manager.cc:1038] Loading token signing keys... I0819 06:34:20.661751 2854 catalog_manager.cc:3605] T 00000000000000000000000000000000 P 280edc58eb9248bf99bb8e42b4eae2b6: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0819 06:34:20.692256 2847 flags.cc:384] Enabled unsafe flag: --enable_leader_failure_detection=false W0819 06:34:20.692656 2847 flags.cc:384] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0819 06:34:20.692855 2847 flags.cc:384] Enabled unsafe flag: --never_fsync=true W0819 06:34:20.706976 2847 flags.cc:384] Enabled experimental flag: --ipki_server_key_size=1024 W0819 06:34:20.707321 2847 flags.cc:384] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.126.1 I0819 06:34:20.727236 2847 tablet_server_main.cc:64] Tablet server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.126.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.4.126.1 --webserver_port=0 --tablet_transaction_memory_limit_mb=-1 --tserver_master_addrs=127.4.126.254:39962 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.2847 --redact=flag --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.126.1 --memory_limit_hard_bytes=4194304 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.5.0-SNAPSHOT revision ba0f21e2316b78f167bbaba44cab392558d97b6f build type FASTDEBUG built by None at 19 Aug 2017 06:31:02 UTC on jenkins-slave-c3oe build id 9422 TSAN enabled I0819 06:34:20.737596 2847 tablet_server_main.cc:71] Initializing tablet server... I0819 06:34:20.739419 2847 system_ntp.cc:96] NTP initialized. Resolution in nanos?: 1 Skew PPM: 500 Current error: 1304405 I0819 06:34:20.740033 2847 env_posix.cc:1586] Not raising process file limit of 65536; it is already as high as it can go I0819 06:34:20.740293 2847 file_cache.cc:464] Constructed file cache lbm with capacity 26214 I0819 06:34:20.745332 2847 server_base.cc:215] Could not load existing FS layout: Not found: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0819 06:34:20.745481 2847 server_base.cc:216] Creating new FS layout I0819 06:34:20.753084 2847 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "4c94be5808b34fc3bb2d82e3988c20f1" format_stamp: "Formatted at 2017-08-19 06:34:20 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:20.753818 2847 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "4c94be5808b34fc3bb2d82e3988c20f1" format_stamp: "Formatted at 2017-08-19 06:34:20 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:20.764747 2847 fs_manager.cc:280] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s I0819 06:34:20.764899 2847 fs_manager.cc:283] Opened local filesystem: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data,/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/wal uuid: "4c94be5808b34fc3bb2d82e3988c20f1" format_stamp: "Formatted at 2017-08-19 06:34:20 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:20.765084 2847 fs_report.cc:345] Block manager report -------------------- 1 data directories: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0819 06:34:20.988761 2847 ts_tablet_manager.cc:190] Loaded tablet metadata (0 live tablets) I0819 06:34:20.990149 2847 tablet_server_main.cc:76] Starting tablet server... I0819 06:34:21.099529 2847 rpc_server.cc:186] RPC server started. Bound to: 127.4.126.1:40980 I0819 06:34:21.100464 2847 webserver.cc:145] Starting webserver on 127.4.126.1:0 I0819 06:34:21.100572 2847 webserver.cc:156] Document root disabled I0819 06:34:21.101963 2847 webserver.cc:278] Webserver started. Bound to: http://127.4.126.1:34635/ I0819 06:34:21.102723 2847 server_base.cc:359] Dumped server information to /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/info.pb I0819 06:34:21.109978 1150 external_mini_cluster.cc:814] Started /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver as pid 2847 I0819 06:34:21.111246 2980 process_memory.cc:172] Process hard memory limit is 0.003906 GB I0819 06:34:21.111263 1150 external_mini_cluster.cc:752] Running /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data --rpc_bind_addresses=127.4.126.2:0 --local_ip_for_outbound_sockets=127.4.126.2 --webserver_interface=127.4.126.2 --webserver_port=0 --tserver_master_addrs=127.4.126.254:39962 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=flag --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --memory_limit_hard_bytes=4194304 --enable_leader_failure_detection=false --tablet_transaction_memory_limit_mb=-1 with env {} I0819 06:34:21.111567 2980 process_memory.cc:174] Process soft memory limit is 0.003125 GB I0819 06:34:21.111709 2980 process_memory.cc:177] Process memory pressure threshold is 0.002344 GB I0819 06:34:21.111343 2847 tablet_server_main.cc:79] Tablet server successfully started. I0819 06:34:21.141036 2979 heartbeater.cc:295] Connected to a master server at 127.4.126.254:39962 I0819 06:34:21.141343 2979 heartbeater.cc:365] Registering TS with master... I0819 06:34:21.141784 2979 heartbeater.cc:392] Master 127.4.126.254:39962 requested a full tablet report, sending... I0819 06:34:21.143374 2813 ts_manager.cc:79] Registered new tserver with Master: 4c94be5808b34fc3bb2d82e3988c20f1 (127.4.126.1:40980) I0819 06:34:21.145021 2813 master_service.cc:197] Signed X509 certificate for tserver {username='slave'} at 127.4.126.1:55824 WARNING: Logging before InitGoogleLogging() is written to STDERR W0819 06:34:21.314486 2983 flags.cc:384] Enabled unsafe flag: --enable_leader_failure_detection=false W0819 06:34:21.315011 2983 flags.cc:384] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0819 06:34:21.315340 2983 flags.cc:384] Enabled unsafe flag: --never_fsync=true W0819 06:34:21.339694 2983 flags.cc:384] Enabled experimental flag: --ipki_server_key_size=1024 W0819 06:34:21.340245 2983 flags.cc:384] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.126.2 I0819 06:34:21.369971 2983 tablet_server_main.cc:64] Tablet server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.126.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.4.126.2 --webserver_port=0 --tablet_transaction_memory_limit_mb=-1 --tserver_master_addrs=127.4.126.254:39962 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.2983 --redact=flag --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.126.2 --memory_limit_hard_bytes=4194304 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.5.0-SNAPSHOT revision ba0f21e2316b78f167bbaba44cab392558d97b6f build type FASTDEBUG built by None at 19 Aug 2017 06:31:02 UTC on jenkins-slave-c3oe build id 9422 TSAN enabled I0819 06:34:21.385112 2983 tablet_server_main.cc:71] Initializing tablet server... I0819 06:34:21.387715 2983 system_ntp.cc:96] NTP initialized. Resolution in nanos?: 1 Skew PPM: 500 Current error: 1304905 I0819 06:34:21.388586 2983 env_posix.cc:1586] Not raising process file limit of 65536; it is already as high as it can go I0819 06:34:21.388993 2983 file_cache.cc:464] Constructed file cache lbm with capacity 26214 I0819 06:34:21.397720 2983 server_base.cc:215] Could not load existing FS layout: Not found: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0819 06:34:21.397938 2983 server_base.cc:216] Creating new FS layout I0819 06:34:21.410368 2983 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "a889c384430446d48d45e519afbee13c" format_stamp: "Formatted at 2017-08-19 06:34:21 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:21.411471 2983 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "a889c384430446d48d45e519afbee13c" format_stamp: "Formatted at 2017-08-19 06:34:21 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:21.427790 2983 fs_manager.cc:280] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s I0819 06:34:21.428005 2983 fs_manager.cc:283] Opened local filesystem: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data,/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/wal uuid: "a889c384430446d48d45e519afbee13c" format_stamp: "Formatted at 2017-08-19 06:34:21 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:21.428275 2983 fs_report.cc:345] Block manager report -------------------- 1 data directories: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0819 06:34:21.654320 2983 ts_tablet_manager.cc:190] Loaded tablet metadata (0 live tablets) I0819 06:34:21.655565 2983 tablet_server_main.cc:76] Starting tablet server... I0819 06:34:21.763267 2983 rpc_server.cc:186] RPC server started. Bound to: 127.4.126.2:45929 I0819 06:34:21.764190 2983 webserver.cc:145] Starting webserver on 127.4.126.2:0 I0819 06:34:21.764308 2983 webserver.cc:156] Document root disabled I0819 06:34:21.765728 2983 webserver.cc:278] Webserver started. Bound to: http://127.4.126.2:57163/ I0819 06:34:21.766502 2983 server_base.cc:359] Dumped server information to /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/info.pb I0819 06:34:21.773919 3105 process_memory.cc:172] Process hard memory limit is 0.003906 GB I0819 06:34:21.774152 3105 process_memory.cc:174] Process soft memory limit is 0.003125 GB I0819 06:34:21.774289 3105 process_memory.cc:177] Process memory pressure threshold is 0.002344 GB I0819 06:34:21.775955 1150 external_mini_cluster.cc:814] Started /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver as pid 2983 I0819 06:34:21.777231 1150 external_mini_cluster.cc:752] Running /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data --rpc_bind_addresses=127.4.126.3:0 --local_ip_for_outbound_sockets=127.4.126.3 --webserver_interface=127.4.126.3 --webserver_port=0 --tserver_master_addrs=127.4.126.254:39962 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=flag --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --memory_limit_hard_bytes=4194304 --enable_leader_failure_detection=false --tablet_transaction_memory_limit_mb=-1 with env {} I0819 06:34:21.796875 3104 heartbeater.cc:295] Connected to a master server at 127.4.126.254:39962 I0819 06:34:21.797222 3104 heartbeater.cc:365] Registering TS with master... I0819 06:34:21.797811 3104 heartbeater.cc:392] Master 127.4.126.254:39962 requested a full tablet report, sending... I0819 06:34:21.799533 2813 ts_manager.cc:79] Registered new tserver with Master: a889c384430446d48d45e519afbee13c (127.4.126.2:45929) I0819 06:34:21.801100 2813 master_service.cc:197] Signed X509 certificate for tserver {username='slave'} at 127.4.126.2:57749 I0819 06:34:21.774013 2983 tablet_server_main.cc:79] Tablet server successfully started. WARNING: Logging before InitGoogleLogging() is written to STDERR W0819 06:34:21.911875 3108 flags.cc:384] Enabled unsafe flag: --enable_leader_failure_detection=false W0819 06:34:21.912271 3108 flags.cc:384] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0819 06:34:21.912511 3108 flags.cc:384] Enabled unsafe flag: --never_fsync=true W0819 06:34:21.926944 3108 flags.cc:384] Enabled experimental flag: --ipki_server_key_size=1024 W0819 06:34:21.927320 3108 flags.cc:384] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.126.3 I0819 06:34:21.947860 3108 tablet_server_main.cc:64] Tablet server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.126.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.4.126.3 --webserver_port=0 --tablet_transaction_memory_limit_mb=-1 --tserver_master_addrs=127.4.126.254:39962 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.3108 --redact=flag --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.126.3 --memory_limit_hard_bytes=4194304 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.5.0-SNAPSHOT revision ba0f21e2316b78f167bbaba44cab392558d97b6f build type FASTDEBUG built by None at 19 Aug 2017 06:31:02 UTC on jenkins-slave-c3oe build id 9422 TSAN enabled I0819 06:34:21.958688 3108 tablet_server_main.cc:71] Initializing tablet server... I0819 06:34:21.960631 3108 system_ntp.cc:96] NTP initialized. Resolution in nanos?: 1 Skew PPM: 500 Current error: 1304905 I0819 06:34:21.961242 3108 env_posix.cc:1586] Not raising process file limit of 65536; it is already as high as it can go I0819 06:34:21.961531 3108 file_cache.cc:464] Constructed file cache lbm with capacity 26214 I0819 06:34:21.966567 3108 server_base.cc:215] Could not load existing FS layout: Not found: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0819 06:34:21.966713 3108 server_base.cc:216] Creating new FS layout I0819 06:34:21.974393 3108 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "8f9cb341a5104891b3f821da4adf01a4" format_stamp: "Formatted at 2017-08-19 06:34:21 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:21.975152 3108 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "8f9cb341a5104891b3f821da4adf01a4" format_stamp: "Formatted at 2017-08-19 06:34:21 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:21.986933 3108 fs_manager.cc:280] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s I0819 06:34:21.987179 3108 fs_manager.cc:283] Opened local filesystem: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data,/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/wal uuid: "8f9cb341a5104891b3f821da4adf01a4" format_stamp: "Formatted at 2017-08-19 06:34:21 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:21.987525 3108 fs_report.cc:345] Block manager report -------------------- 1 data directories: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0819 06:34:22.147454 2979 heartbeater.cc:384] Master 127.4.126.254:39962 was elected leader, sending a full tablet report... I0819 06:34:22.164096 3108 ts_tablet_manager.cc:190] Loaded tablet metadata (0 live tablets) I0819 06:34:22.165225 3108 tablet_server_main.cc:76] Starting tablet server... I0819 06:34:22.257033 3108 rpc_server.cc:186] RPC server started. Bound to: 127.4.126.3:38257 I0819 06:34:22.257912 3108 webserver.cc:145] Starting webserver on 127.4.126.3:0 I0819 06:34:22.258013 3108 webserver.cc:156] Document root disabled I0819 06:34:22.259202 3108 webserver.cc:278] Webserver started. Bound to: http://127.4.126.3:50294/ I0819 06:34:22.260020 3108 server_base.cc:359] Dumped server information to /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/info.pb I0819 06:34:22.262945 3230 process_memory.cc:172] Process hard memory limit is 0.003906 GB I0819 06:34:22.263499 3230 process_memory.cc:174] Process soft memory limit is 0.003125 GB I0819 06:34:22.263629 3230 process_memory.cc:177] Process memory pressure threshold is 0.002344 GB I0819 06:34:22.267308 1150 external_mini_cluster.cc:814] Started /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver as pid 3108 I0819 06:34:22.263447 3108 tablet_server_main.cc:79] Tablet server successfully started. I0819 06:34:22.308485 3229 heartbeater.cc:295] Connected to a master server at 127.4.126.254:39962 I0819 06:34:22.308759 3229 heartbeater.cc:365] Registering TS with master... I0819 06:34:22.309196 3229 heartbeater.cc:392] Master 127.4.126.254:39962 requested a full tablet report, sending... I0819 06:34:22.310469 2813 ts_manager.cc:79] Registered new tserver with Master: 8f9cb341a5104891b3f821da4adf01a4 (127.4.126.3:38257) I0819 06:34:22.311771 2813 master_service.cc:197] Signed X509 certificate for tserver {username='slave'} at 127.4.126.3:48002 I0819 06:34:22.312984 1150 external_mini_cluster.cc:454] 3 TS(s) registered with all masters I0819 06:34:22.331372 2812 catalog_manager.cc:1281] CreateTable from {username='slave'} at 127.0.0.1:58432: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0819 06:34:22.370795 3166 tablet_service.cc:684] Processing CreateTablet for tablet 31d2fe4307b848379aea3d36390148e4 (table=TestTable [id=1847cb67198a4de9b80681d881eac0ec]), partition=RANGE (key) PARTITION UNBOUNDED I0819 06:34:22.373167 3041 tablet_service.cc:684] Processing CreateTablet for tablet 31d2fe4307b848379aea3d36390148e4 (table=TestTable [id=1847cb67198a4de9b80681d881eac0ec]), partition=RANGE (key) PARTITION UNBOUNDED I0819 06:34:22.376530 2916 tablet_service.cc:684] Processing CreateTablet for tablet 31d2fe4307b848379aea3d36390148e4 (table=TestTable [id=1847cb67198a4de9b80681d881eac0ec]), partition=RANGE (key) PARTITION UNBOUNDED I0819 06:34:22.385582 3041 ts_tablet_manager.cc:877] T 31d2fe4307b848379aea3d36390148e4 P a889c384430446d48d45e519afbee13c: Registered tablet (data state: TABLET_DATA_READY) I0819 06:34:22.387243 3166 ts_tablet_manager.cc:877] T 31d2fe4307b848379aea3d36390148e4 P 8f9cb341a5104891b3f821da4adf01a4: Registered tablet (data state: TABLET_DATA_READY) I0819 06:34:22.392791 2916 ts_tablet_manager.cc:877] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Registered tablet (data state: TABLET_DATA_READY) I0819 06:34:22.395138 3250 ts_tablet_manager.cc:740] T 31d2fe4307b848379aea3d36390148e4 P 8f9cb341a5104891b3f821da4adf01a4: Bootstrapping tablet I0819 06:34:22.395668 3250 tablet_bootstrap.cc:402] T 31d2fe4307b848379aea3d36390148e4 P 8f9cb341a5104891b3f821da4adf01a4: Bootstrap starting. I0819 06:34:22.395248 3249 ts_tablet_manager.cc:740] T 31d2fe4307b848379aea3d36390148e4 P a889c384430446d48d45e519afbee13c: Bootstrapping tablet I0819 06:34:22.396440 3249 tablet_bootstrap.cc:402] T 31d2fe4307b848379aea3d36390148e4 P a889c384430446d48d45e519afbee13c: Bootstrap starting. I0819 06:34:22.397202 3251 ts_tablet_manager.cc:740] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Bootstrapping tablet I0819 06:34:22.397675 3251 tablet_bootstrap.cc:402] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Bootstrap starting. I0819 06:34:22.398980 3250 tablet_bootstrap.cc:537] T 31d2fe4307b848379aea3d36390148e4 P 8f9cb341a5104891b3f821da4adf01a4: No blocks or log segments found. Creating new log. I0819 06:34:22.399101 3249 tablet_bootstrap.cc:537] T 31d2fe4307b848379aea3d36390148e4 P a889c384430446d48d45e519afbee13c: No blocks or log segments found. Creating new log. I0819 06:34:22.400344 3249 log.cc:511] T 31d2fe4307b848379aea3d36390148e4 P a889c384430446d48d45e519afbee13c: Log is configured to *not* fsync() on all Append() calls I0819 06:34:22.400689 3250 log.cc:511] T 31d2fe4307b848379aea3d36390148e4 P 8f9cb341a5104891b3f821da4adf01a4: Log is configured to *not* fsync() on all Append() calls I0819 06:34:22.400946 3251 tablet_bootstrap.cc:537] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: No blocks or log segments found. Creating new log. I0819 06:34:22.402493 3251 log.cc:511] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Log is configured to *not* fsync() on all Append() calls I0819 06:34:22.403735 3250 tablet_bootstrap.cc:402] T 31d2fe4307b848379aea3d36390148e4 P 8f9cb341a5104891b3f821da4adf01a4: No bootstrap required, opened a new log I0819 06:34:22.404132 3250 ts_tablet_manager.cc:752] T 31d2fe4307b848379aea3d36390148e4 P 8f9cb341a5104891b3f821da4adf01a4: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s I0819 06:34:22.405748 3251 tablet_bootstrap.cc:402] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: No bootstrap required, opened a new log I0819 06:34:22.406196 3251 ts_tablet_manager.cc:752] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s I0819 06:34:22.406311 3249 tablet_bootstrap.cc:402] T 31d2fe4307b848379aea3d36390148e4 P a889c384430446d48d45e519afbee13c: No bootstrap required, opened a new log I0819 06:34:22.405932 3250 raft_consensus.cc:286] T 31d2fe4307b848379aea3d36390148e4 P 8f9cb341a5104891b3f821da4adf01a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c94be5808b34fc3bb2d82e3988c20f1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 40980 } } peers { permanent_uuid: "a889c384430446d48d45e519afbee13c" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 45929 } } peers { permanent_uuid: "8f9cb341a5104891b3f821da4adf01a4" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 38257 } } I0819 06:34:22.406728 3249 ts_tablet_manager.cc:752] T 31d2fe4307b848379aea3d36390148e4 P a889c384430446d48d45e519afbee13c: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s I0819 06:34:22.406891 3250 raft_consensus.cc:574] T 31d2fe4307b848379aea3d36390148e4 P 8f9cb341a5104891b3f821da4adf01a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9cb341a5104891b3f821da4adf01a4, State: Running, Role: FOLLOWER I0819 06:34:22.407258 3250 consensus_queue.cc:190] T 31d2fe4307b848379aea3d36390148e4 P 8f9cb341a5104891b3f821da4adf01a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0819 06:34:22.407795 3251 raft_consensus.cc:286] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c94be5808b34fc3bb2d82e3988c20f1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 40980 } } peers { permanent_uuid: "a889c384430446d48d45e519afbee13c" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 45929 } } peers { permanent_uuid: "8f9cb341a5104891b3f821da4adf01a4" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 38257 } } I0819 06:34:22.408411 3251 raft_consensus.cc:574] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c94be5808b34fc3bb2d82e3988c20f1, State: Running, Role: FOLLOWER I0819 06:34:22.408643 3251 consensus_queue.cc:190] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0819 06:34:22.408493 3249 raft_consensus.cc:286] T 31d2fe4307b848379aea3d36390148e4 P a889c384430446d48d45e519afbee13c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c94be5808b34fc3bb2d82e3988c20f1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 40980 } } peers { permanent_uuid: "a889c384430446d48d45e519afbee13c" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 45929 } } peers { permanent_uuid: "8f9cb341a5104891b3f821da4adf01a4" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 38257 } } I0819 06:34:22.409286 3249 raft_consensus.cc:574] T 31d2fe4307b848379aea3d36390148e4 P a889c384430446d48d45e519afbee13c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a889c384430446d48d45e519afbee13c, State: Running, Role: FOLLOWER I0819 06:34:22.409565 3249 consensus_queue.cc:190] T 31d2fe4307b848379aea3d36390148e4 P a889c384430446d48d45e519afbee13c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0819 06:34:22.411924 3229 heartbeater.cc:384] Master 127.4.126.254:39962 was elected leader, sending a full tablet report... I0819 06:34:22.412334 3250 ts_tablet_manager.cc:781] T 31d2fe4307b848379aea3d36390148e4 P 8f9cb341a5104891b3f821da4adf01a4: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s I0819 06:34:22.412843 3251 ts_tablet_manager.cc:781] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s I0819 06:34:22.413753 3104 heartbeater.cc:384] Master 127.4.126.254:39962 was elected leader, sending a full tablet report... I0819 06:34:22.414816 3249 ts_tablet_manager.cc:781] T 31d2fe4307b848379aea3d36390148e4 P a889c384430446d48d45e519afbee13c: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s I0819 06:34:22.424657 1150 external_mini_cluster.cc:454] 3 TS(s) registered with all masters I0819 06:34:22.426683 1150 ts_itest-base.h:198] Waiting for 1 tablets on tserver 4c94be5808b34fc3bb2d82e3988c20f1 to finish bootstrapping I0819 06:34:22.435641 1150 ts_itest-base.h:198] Waiting for 1 tablets on tserver a889c384430446d48d45e519afbee13c to finish bootstrapping I0819 06:34:22.444875 1150 ts_itest-base.h:198] Waiting for 1 tablets on tserver 8f9cb341a5104891b3f821da4adf01a4 to finish bootstrapping I0819 06:34:22.481040 3186 raft_consensus.cc:413] T 31d2fe4307b848379aea3d36390148e4 P 8f9cb341a5104891b3f821da4adf01a4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request) I0819 06:34:22.481266 3186 raft_consensus.cc:2448] T 31d2fe4307b848379aea3d36390148e4 P 8f9cb341a5104891b3f821da4adf01a4 [term 0 FOLLOWER]: Advancing to term 1 I0819 06:34:22.484148 3186 raft_consensus.cc:437] T 31d2fe4307b848379aea3d36390148e4 P 8f9cb341a5104891b3f821da4adf01a4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c94be5808b34fc3bb2d82e3988c20f1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 40980 } } peers { permanent_uuid: "a889c384430446d48d45e519afbee13c" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 45929 } } peers { permanent_uuid: "8f9cb341a5104891b3f821da4adf01a4" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 38257 } } I0819 06:34:22.485240 3186 leader_election.cc:217] T 31d2fe4307b848379aea3d36390148e4 P 8f9cb341a5104891b3f821da4adf01a4 [CANDIDATE]: Term 1 election: Requesting vote from peer 4c94be5808b34fc3bb2d82e3988c20f1 I0819 06:34:22.485711 3186 leader_election.cc:217] T 31d2fe4307b848379aea3d36390148e4 P 8f9cb341a5104891b3f821da4adf01a4 [CANDIDATE]: Term 1 election: Requesting vote from peer a889c384430446d48d45e519afbee13c I0819 06:34:22.497421 2936 raft_consensus.cc:2448] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1 [term 0 FOLLOWER]: Advancing to term 1 I0819 06:34:22.500448 2936 raft_consensus.cc:1967] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9cb341a5104891b3f821da4adf01a4 in term 1. I0819 06:34:22.501493 3121 leader_election.cc:349] T 31d2fe4307b848379aea3d36390148e4 P 8f9cb341a5104891b3f821da4adf01a4 [CANDIDATE]: Term 1 election: Vote granted by peer 4c94be5808b34fc3bb2d82e3988c20f1 I0819 06:34:22.501677 3121 leader_election.cc:244] T 31d2fe4307b848379aea3d36390148e4 P 8f9cb341a5104891b3f821da4adf01a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0819 06:34:22.502357 3255 raft_consensus.cc:2205] T 31d2fe4307b848379aea3d36390148e4 P 8f9cb341a5104891b3f821da4adf01a4 [term 1 FOLLOWER]: Leader election won for term 1 I0819 06:34:22.502862 3061 raft_consensus.cc:2448] T 31d2fe4307b848379aea3d36390148e4 P a889c384430446d48d45e519afbee13c [term 0 FOLLOWER]: Advancing to term 1 I0819 06:34:22.506224 3061 raft_consensus.cc:1967] T 31d2fe4307b848379aea3d36390148e4 P a889c384430446d48d45e519afbee13c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9cb341a5104891b3f821da4adf01a4 in term 1. I0819 06:34:22.507299 3119 leader_election.cc:349] T 31d2fe4307b848379aea3d36390148e4 P 8f9cb341a5104891b3f821da4adf01a4 [CANDIDATE]: Term 1 election: Vote granted by peer a889c384430446d48d45e519afbee13c I0819 06:34:22.508108 3255 raft_consensus.cc:540] T 31d2fe4307b848379aea3d36390148e4 P 8f9cb341a5104891b3f821da4adf01a4 [term 1 LEADER]: Becoming Leader. State: Replica: 8f9cb341a5104891b3f821da4adf01a4, State: Running, Role: LEADER I0819 06:34:22.508594 3255 consensus_queue.cc:169] T 31d2fe4307b848379aea3d36390148e4 P 8f9cb341a5104891b3f821da4adf01a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c94be5808b34fc3bb2d82e3988c20f1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 40980 } } peers { permanent_uuid: "a889c384430446d48d45e519afbee13c" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 45929 } } peers { permanent_uuid: "8f9cb341a5104891b3f821da4adf01a4" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 38257 } } I0819 06:34:22.523097 1150 cluster_itest_util.cc:220] Not converged past 1 yet: 0.0 0.0 1.1 I0819 06:34:22.524255 2813 catalog_manager.cc:2559] T 31d2fe4307b848379aea3d36390148e4 reported consensus state change: term changed from 0 to 1, leader changed from to 8f9cb341a5104891b3f821da4adf01a4 (127.4.126.3). New consensus state: current_term: 1 leader_uuid: "8f9cb341a5104891b3f821da4adf01a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c94be5808b34fc3bb2d82e3988c20f1" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 40980 } } peers { permanent_uuid: "a889c384430446d48d45e519afbee13c" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 45929 } } peers { permanent_uuid: "8f9cb341a5104891b3f821da4adf01a4" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 38257 } } } I0819 06:34:22.628089 1150 cluster_itest_util.cc:220] Not converged past 1 yet: 0.0 0.0 1.1 I0819 06:34:22.832985 1150 cluster_itest_util.cc:220] Not converged past 1 yet: 0.0 0.0 1.1 I0819 06:34:22.922474 2936 raft_consensus.cc:961] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1 [term 1 FOLLOWER]: Refusing update from remote peer 8f9cb341a5104891b3f821da4adf01a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0819 06:34:22.923779 3255 consensus_queue.cc:726] T 31d2fe4307b848379aea3d36390148e4 P 8f9cb341a5104891b3f821da4adf01a4 [LEADER]: Connected to new peer: Peer: 4c94be5808b34fc3bb2d82e3988c20f1, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I0819 06:34:22.941783 3061 raft_consensus.cc:961] T 31d2fe4307b848379aea3d36390148e4 P a889c384430446d48d45e519afbee13c [term 1 FOLLOWER]: Refusing update from remote peer 8f9cb341a5104891b3f821da4adf01a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0819 06:34:22.942854 3261 consensus_queue.cc:726] T 31d2fe4307b848379aea3d36390148e4 P 8f9cb341a5104891b3f821da4adf01a4 [LEADER]: Connected to new peer: Peer: a889c384430446d48d45e519afbee13c, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I0819 06:34:23.138031 1150 external_mini_cluster.cc:944] Killing /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver with pid 2983 W0819 06:34:23.155908 2784 connection.cc:551] client connection to 127.4.126.2:45929 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:23.155972 2796 connection.cc:551] client connection to 127.4.126.2:45929 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:23.155971 2798 connection.cc:551] server connection from 127.4.126.2:57749 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:23.157076 2782 connection.cc:551] client connection to 127.4.126.2:45929 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:23.157125 3119 connection.cc:551] client connection to 127.4.126.2:45929 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0819 06:34:23.158179 1150 external_mini_cluster.cc:944] Killing /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver with pid 3108 W0819 06:34:23.175839 2795 connection.cc:551] client connection to 127.4.126.3:38257 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:23.175896 2784 connection.cc:551] client connection to 127.4.126.3:38257 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:23.176807 2796 connection.cc:551] server connection from 127.4.126.3:48002 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:23.176795 2781 connection.cc:551] client connection to 127.4.126.3:38257 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:23.176842 2869 connection.cc:551] server connection from 127.4.126.3:60217 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0819 06:34:27.584004 2980 maintenance_manager.cc:391] P 4c94be5808b34fc3bb2d82e3988c20f1: we have exceeded our soft memory limit (current capacity is 184.55%). However, there are no ops currently runnable which would free memory. I0819 06:34:27.898823 2980 maintenance_manager.cc:391] P 4c94be5808b34fc3bb2d82e3988c20f1: we have exceeded our soft memory limit (current capacity is 184.55%). However, there are no ops currently runnable which would free memory. I0819 06:34:28.210038 2980 maintenance_manager.cc:391] P 4c94be5808b34fc3bb2d82e3988c20f1: we have exceeded our soft memory limit (current capacity is 184.55%). However, there are no ops currently runnable which would free memory. W0819 06:34:28.219013 3269 log.cc:665] Time spent T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Append to log took a long time: real 0.096s user 0.092s sys 0.000s I0819 06:34:28.559247 2980 maintenance_manager.cc:391] P 4c94be5808b34fc3bb2d82e3988c20f1: we have exceeded our soft memory limit (current capacity is 184.55%). However, there are no ops currently runnable which would free memory. I0819 06:34:28.906147 2980 maintenance_manager.cc:391] P 4c94be5808b34fc3bb2d82e3988c20f1: we have exceeded our soft memory limit (current capacity is 184.55%). However, there are no ops currently runnable which would free memory. I0819 06:34:28.280356 2936 rpcz_store.cc:249] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49244 (request call id 5) took 4320ms. Request Metrics: {"spinlock_wait_cycles":401206272,"thread_start_us":929,"threads_started":1,"wal-append.queue_time_us":1060,"child_traces":[["txn",{"num_ops":1,"prepare.queue_time_us":897,"prepare.run_cpu_time_us":888,"prepare.run_wall_time_us":888,"thread_start_us":756,"threads_started":1}],["txn",{"num_ops":1,"prepare.queue_time_us":671,"prepare.run_cpu_time_us":270,"prepare.run_wall_time_us":270}],["txn",{"num_ops":1,"prepare.queue_time_us":748,"prepare.run_cpu_time_us":232,"prepare.run_wall_time_us":232}],["txn",{"num_ops":1,"prepare.queue_time_us":831,"prepare.run_cpu_time_us":265,"prepare.run_wall_time_us":265}],["txn",{"num_ops":1,"prepare.queue_time_us":733,"prepare.run_cpu_time_us":237,"prepare.run_wall_time_us":237}],["txn",{"num_ops":1,"prepare.queue_time_us":802,"prepare.run_cpu_time_us":228,"prepare.run_wall_time_us":227}],["txn",{"mutex_wait_us":20,"num_ops":1,"prepare.queue_time_us":894,"prepare.run_cpu_time_us":238,"prepare.run_wall_time_us":238}],["txn",{"mutex_wait_us":14,"num_ops":1,"prepare.queue_time_us":924,"prepare.run_cpu_time_us":244,"prepare.run_wall_time_us":243}],["txn",{"num_ops":1,"prepare.queue_time_us":1019,"prepare.run_cpu_time_us":250,"prepare.run_wall_time_us":249}],["txn",{"num_ops":1,"prepare.queue_time_us":1122,"prepare.run_cpu_time_us":234,"prepare.run_wall_time_us":234}],["txn",{"mutex_wait_us":25,"num_ops":1,"prepare.queue_time_us":1213,"prepare.run_cpu_time_us":253,"prepare.run_wall_time_us":253}],["txn",{"num_ops":1,"prepare.queue_time_us":1286,"prepare.run_cpu_time_us":237,"prepare.run_wall_time_us":238}],["txn",{"num_ops":1,"prepare.queue_time_us":1379,"prepare.run_cpu_time_us":232,"prepare.run_wall_time_us":232}],["txn",{"num_ops":1,"prepare.queue_time_us":1474,"prepare.run_cpu_time_us":244,"prepare.run_wall_time_us":243}],["txn",{"num_ops":1,"prepare.queue_time_us":1542,"prepare.run_cpu_time_us":243,"prepare.run_wall_time_us":243}],["txn",{"num_ops":1,"prepare.queue_time_us":1551,"prepare.run_cpu_time_us":229,"prepare.run_wall_time_us":229}],["txn",{"num_ops":1,"prepare.queue_time_us":1604,"prepare.run_cpu_time_us":267,"prepare.run_wall_time_us":267}],["txn",{"mutex_wait_us":16,"num_ops":1,"prepare.queue_time_us":1746,"prepare.run_cpu_time_us":350,"prepare.run_wall_time_us":350}],["txn",{"num_ops":1,"prepare.queue_time_us":1941,"prepare.run_cpu_time_us":249,"prepare.run_wall_time_us":249}],["txn",{"num_ops":1,"prepare.queue_time_us":2045,"prepare.run_cpu_time_us":237,"prepare.run_wall_time_us":237}],["txn",{"num_ops":1,"prepare.queue_time_us":2134,"prepare.run_cpu_time_us":273,"prepare.run_wall_time_us":272}],["txn",{"num_ops":1,"prepare.queue_time_us":2254,"prepare.run_cpu_time_us":248,"prepare.run_wall_time_us":253}],["txn",{"num_ops":1,"prepare.queue_time_us":2366,"prepare.run_cpu_time_us":222,"prepare.run_wall_time_us":222}],["txn",{"num_ops":1,"prepare.queue_time_us":2419,"prepare.run_cpu_time_us":229,"prepare.run_wall_time_us":230}],["txn",{"num_ops":1,"prepare.queue_time_us":2491,"prepare.run_cpu_time_us":232,"prepare.run_wall_time_us":232}],["txn",{"num_ops":1,"prepare.queue_time_us":2578,"prepare.run_cpu_time_us":225,"prepare.run_wall_time_us":225}],["txn",{"num_ops":1,"prepare.queue_time_us":2663,"prepare.run_cpu_time_us":227,"prepare.run_wall_time_us":226}],["txn",{"num_ops":1,"prepare.queue_time_us":2709,"prepare.run_cpu_time_us":247,"prepare.run_wall_time_us":246}],["txn",{"mutex_wait_us":18,"num_ops":1,"prepare.queue_time_us":2827,"prepare.run_cpu_time_us":230,"prepare.run_wall_time_us":230}],["txn",{"num_ops":1,"prepare.queue_time_us":2937,"prepare.run_cpu_time_us":233,"prepare.run_wall_time_us":233}],["txn",{"mutex_wait_us":20,"num_ops":1,"prepare.queue_time_us":3030,"prepare.run_cpu_time_us":268,"prepare.run_wall_time_us":268}],["txn",{"mutex_wait_us":24,"num_ops":1,"prepare.queue_time_us":3191,"prepare.run_cpu_time_us":265,"prepare.run_wall_time_us":264}],["txn",{"num_ops":1,"prepare.queue_time_us":3204,"prepare.run_cpu_time_us":294,"prepare.run_wall_time_us":294}],["txn",{"num_ops":1,"prepare.queue_time_us":3138,"prepare.run_cpu_time_us":296,"prepare.run_wall_time_us":296}],["txn",{"mutex_wait_us":2,"num_ops":1,"prepare.queue_time_us":3285,"prepare.run_cpu_time_us":255,"prepare.run_wall_time_us":255}],["txn",{"num_ops":1,"prepare.queue_time_us":3391,"prepare.run_cpu_time_us":235,"prepare.run_wall_time_us":235}],["txn",{"num_ops":1,"prepare.queue_time_us":3494,"prepare.run_cpu_time_us":228,"prepare.run_wall_time_us":228}],["txn",{"num_ops":1,"prepare.queue_time_us":3586,"prepare.run_cpu_time_us":231,"prepare.run_wall_time_us":231}],["txn",{"num_ops":1,"prepare.queue_time_us":3673,"prepare.run_cpu_time_us":225,"prepare.run_wall_time_us":224}],["txn",{"num_ops":1,"prepare.queue_time_us":3756,"prepare.run_cpu_time_us":230,"prepare.run_wall_time_us":230}],["txn",{"num_ops":1,"prepare.queue_time_us":3851,"prepare.run_cpu_time_us":234,"prepare.run_wall_time_us":235}],["txn",{"num_ops":1,"prepare.queue_time_us":3949,"prepare.run_cpu_time_us":263,"prepare.run_wall_time_us":276}],["txn",{"num_ops":1,"prepare.queue_time_us":4098,"prepare.run_cpu_time_us":245,"prepare.run_wall_time_us":244}],["txn",{"num_ops":1,"prepare.queue_time_us":4204,"prepare.run_cpu_time_us":457,"prepare.run_wall_time_us":457}],["txn",{"num_ops":1,"prepare.queue_time_us":4501,"prepare.run_cpu_time_us":229,"prepare.run_wall_time_us":229}],["txn",{"num_ops":1,"prepare.queue_time_us":4596,"prepare.run_cpu_time_us":233,"prepare.run_wall_time_us":233}],["txn",{"num_ops":1,"prepare.queue_time_us":4642,"prepare.run_cpu_time_us":246,"prepare.run_wall_time_us":245}],["txn",{"num_ops":1,"prepare.queue_time_us":4726,"prepare.run_cpu_time_us":253,"prepare.run_wall_time_us":253}],["txn",{"num_ops":1,"prepare.queue_time_us":4830,"prepare.run_cpu_time_us":241,"prepare.run_wall_time_us":240}],["txn",{"num_ops":1,"prepare.queue_time_us":4925,"prepare.run_cpu_time_us":258,"prepare.run_wall_time_us":258}],["txn",{"num_ops":1,"prepare.queue_time_us":5072,"prepare.run_cpu_time_us":368,"prepare.run_wall_time_us":368}],["txn",{"num_ops":1,"prepare.queue_time_us":5246,"prepare.run_cpu_time_us":353,"prepare.run_wall_time_us":352}],["txn",{"num_ops":1,"prepare.queue_time_us":5440,"prepare.run_cpu_time_us":307,"prepare.run_wall_time_us":308}],["txn",{"num_ops":1,"prepare.queue_time_us":5616,"prepare.run_cpu_time_us":245,"prepare.run_wall_time_us":247}],["txn",{"num_ops":1,"prepare.queue_time_us":5720,"prepare.run_cpu_time_us":238,"prepare.run_wall_time_us":237}],["txn",{"num_ops":1,"prepare.queue_time_us":5825,"prepare.run_cpu_time_us":231,"prepare.run_wall_time_us":230}],["txn",{"num_ops":1,"prepare.queue_time_us":5922,"prepare.run_cpu_time_us":241,"prepare.run_wall_time_us":240}],["txn",{"num_ops":1,"prepare.queue_time_us":6035,"prepare.run_cpu_time_us":229,"prepare.run_wall_time_us":229}],["txn",{"num_ops":1,"prepare.queue_time_us":6128,"prepare.run_cpu_time_us":249,"prepare.run_wall_time_us":249}],["txn",{"num_ops":1,"prepare.queue_time_us":6270,"prepare.run_cpu_time_us":261,"prepare.run_wall_time_us":261}],["txn",{"mutex_wait_us":2,"num_ops":1,"prepare.queue_time_us":6410,"prepare.run_cpu_time_us":242,"prepare.run_wall_time_us":242}],["txn",{"num_ops":1,"prepare.queue_time_us":6488,"prepare.run_cpu_time_us":231,"prepare.run_wall_time_us":231}],["txn",{"num_ops":1,"prepare.queue_time_us":6573,"prepare.run_cpu_time_us":242,"prepare.run_wall_time_us":242}],["txn",{"num_ops":1,"prepare.queue_time_us":6614,"prepare.run_cpu_time_us":239,"prepare.run_wall_time_us":238}],["txn",{"num_ops":1,"prepare.queue_time_us":6574,"prepare.run_cpu_time_us":289,"prepare.run_wall_time_us":293}],["txn",{"num_ops":1,"prepare.queue_time_us":6577,"prepare.run_cpu_time_us":233,"prepare.run_wall_time_us":233}],["txn",{"num_ops":1,"prepare.queue_time_us":6661,"prepare.run_cpu_time_us":245,"prepare.run_wall_time_us":244}],["txn",{"num_ops":1,"prepare.queue_time_us":6719,"prepare.run_cpu_time_us":234,"prepare.run_wall_time_us":234}],["txn",{"num_ops":1,"prepare.queue_time_us":6810,"prepare.run_cpu_time_us":228,"prepare.run_wall_time_us":228}],["txn",{"num_ops":1,"prepare.queue_time_us":6868,"prepare.run_cpu_time_us":224,"prepare.run_wall_time_us":224}],["txn",{"num_ops":1,"prepare.queue_time_us":6929,"prepare.run_cpu_time_us":233,"prepare.run_wall_time_us":232}],["txn",{"num_ops":1,"prepare.queue_time_us":7022,"prepare.run_cpu_time_us":223,"prepare.run_wall_time_us":223}],["txn",{"mutex_wait_us":16,"num_ops":1,"prepare.queue_time_us":7105,"prepare.run_cpu_time_us":227,"prepare.run_wall_time_us":227}],["txn",{"num_ops":1,"prepare.queue_time_us":7186,"prepare.run_cpu_time_us":244,"prepare.run_wall_time_us":244}],["txn",{"num_ops":1,"prepare.queue_time_us":7299,"prepare.run_cpu_time_us":236,"prepare.run_wall_time_us":235}],["txn",{"num_ops":1,"prepare.queue_time_us":7005,"prepare.run_cpu_time_us":239,"prepare.run_wall_time_us":239}],["txn",{"num_ops":1,"prepare.queue_time_us":7077,"prepare.run_cpu_time_us":237,"prepare.run_wall_time_us":237}],["txn",{"num_ops":1,"prepare.queue_time_us":7157,"prepare.run_cpu_time_us":247,"prepare.run_wall_time_us":247}],["txn",{"num_ops":1,"prepare.queue_time_us":7261,"prepare.run_cpu_time_us":235,"prepare.run_wall_time_us":235}],["txn",{"num_ops":1,"prepare.queue_time_us":7365,"prepare.run_cpu_time_us":230,"prepare.run_wall_time_us":230}],["txn",{"num_ops":1,"prepare.queue_time_us":7451,"prepare.run_cpu_time_us":227,"prepare.run_wall_time_us":227}],["txn",{"num_ops":1,"prepare.queue_time_us":7527,"prepare.run_cpu_time_us":243,"prepare.run_wall_time_us":245}],["txn",{"num_ops":1,"prepare.queue_time_us":7629,"prepare.run_cpu_time_us":224,"prepare.run_wall_time_us":225}],["txn",{"num_ops":1,"prepare.queue_time_us":7714,"prepare.run_cpu_time_us":241,"prepare.run_wall_time_us":241}],["txn",{"num_ops":1,"prepare.queue_time_us":7819,"prepare.run_cpu_time_us":236,"prepare.run_wall_time_us":237}],["txn",{"mutex_wait_us":3,"num_ops":1,"prepare.queue_time_us":7900,"prepare.run_cpu_time_us":264,"prepare.run_wall_time_us":263}],["txn",{"num_ops":1,"prepare.queue_time_us":8031,"prepare.run_cpu_time_us":232,"prepare.run_wall_time_us":233}],["txn",{"num_ops":1,"prepare.queue_time_us":8075,"prepare.run_cpu_time_us":246,"prepare.run_wall_time_us":246}],["txn",{"mutex_wait_us":24,"num_ops":1,"prepare.queue_time_us":8139,"prepare.run_cpu_time_us":237,"prepare.run_wall_time_us":237}],["txn",{"mutex_wait_us":30,"num_ops":1,"prepare.queue_time_us":8244,"prepare.run_cpu_time_us":237,"prepare.run_wall_time_us":238}],["txn",{"num_ops":1,"prepare.queue_time_us":8287,"prepare.run_cpu_time_us":238,"prepare.run_wall_time_us":238}],["txn",{"num_ops":1,"prepare.queue_time_us":8390,"prepare.run_cpu_time_us":231,"prepare.run_wall_time_us":230}],["txn",{"num_ops":1,"prepare.queue_time_us":8256,"prepare.run_cpu_time_us":272,"prepare.run_wall_time_us":272}],["txn",{"num_ops":1,"prepare.queue_time_us":8439,"prepare.run_cpu_time_us":255,"prepare.run_wall_time_us":254}],["txn",{"num_ops":1,"prepare.queue_time_us":8556,"prepare.run_cpu_time_us":227,"prepare.run_wall_time_us":226}],["txn",{"num_ops":1,"prepare.queue_time_us":8633,"prepare.run_cpu_time_us":252,"prepare.run_wall_time_us":252}],["txn",{"num_ops":1,"prepare.queue_time_us":8747,"prepare.run_cpu_time_us":276,"prepare.run_wall_time_us":276}],["txn",{"num_ops":1,"prepare.queue_time_us":8843,"prepare.run_cpu_time_us":235,"prepare.run_wall_time_us":235}],["txn",{"num_ops":1,"prepare.queue_time_us":8929,"prepare.run_cpu_time_us":254,"prepare.run_wall_time_us":254}],["txn",{"num_ops":1,"prepare.queue_time_us":9041,"prepare.run_cpu_time_us":242,"prepare.run_wall_time_us":242}],["txn",{"num_ops":1,"prepare.queue_time_us":9139,"prepare.run_cpu_time_us":232,"prepare.run_wall_time_us":232}],["txn",{"num_ops":1,"prepare.queue_time_us":9207,"prepare.run_cpu_time_us":243,"prepare.run_wall_time_us":243}],["txn",{"num_ops":1,"prepare.queue_time_us":9312,"prepare.run_cpu_time_us":226,"prepare.run_wall_time_us":226}],["txn",{"num_ops":1,"prepare.queue_time_us":9405,"prepare.run_cpu_time_us":221,"prepare.run_wall_time_us":222}],["txn",{"num_ops":1,"prepare.queue_time_us":9482,"prepare.run_cpu_time_us":240,"prepare.run_wall_time_us":245}],["txn",{"num_ops":1,"prepare.queue_time_us":9591,"prepare.run_cpu_time_us":230,"prepare.run_wall_time_us":231}],["txn",{"num_ops":1,"prepare.queue_time_us":9690,"prepare.run_cpu_time_us":226,"prepare.run_wall_time_us":226}],["txn",{"num_ops":1,"prepare.queue_time_us":9768,"prepare.run_cpu_time_us":221,"prepare.run_wall_time_us":221}],["txn",{"num_ops":1,"prepare.queue_time_us":9845,"prepare.run_cpu_time_us":223,"prepare.run_wall_time_us":223}],["txn",{"num_ops":1,"prepare.queue_time_us":9919,"prepare.run_cpu_time_us":226,"prepare.run_wall_time_us":227}],["txn",{"num_ops":1,"prepare.queue_time_us":10000,"prepare.run_cpu_time_us":237,"prepare.run_wall_time_us":236}],["txn",{"num_ops":1,"prepare.queue_time_us":10111,"prepare.run_cpu_time_us":225,"prepare.run_wall_time_us":225}],["txn",{"num_ops":1,"prepare.queue_time_us":10198,"prepare.run_cpu_time_us":245,"prepare.run_wall_time_us":244}],["txn",{"num_ops":1,"prepare.queue_time_us":10303,"prepare.run_cpu_time_us":227,"prepare.run_wall_time_us":227}],["txn",{"num_ops":1,"prepare.queue_time_us":10400,"prepare.run_cpu_time_us":240,"prepare.run_wall_time_us":239}],["txn",{"num_ops":1,"prepare.queue_time_us":10475,"prepare.run_cpu_time_us":246,"prepare.run_wall_time_us":246}],["txn",{"num_ops":1,"prepare.queue_time_us":10572,"prepare.run_cpu_time_us":240,"prepare.run_wall_time_us":240}],["txn",{"num_ops":1,"prepare.queue_time_us":10638,"prepare.run_cpu_time_us":228,"prepare.run_wall_time_us":228}],["txn",{"num_ops":1,"prepare.queue_time_us":10646,"prepare.run_cpu_time_us":224,"prepare.run_wall_time_us":224}],["txn",{"num_ops":1,"prepare.queue_time_us":10737,"prepare.run_cpu_time_us":221,"prepare.run_wall_time_us":221}],["txn",{"num_ops":1,"prepare.queue_time_us":10831,"prepare.run_cpu_time_us":222,"prepare.run_wall_time_us":222}],["txn",{"num_ops":1,"prepare.queue_time_us":10907,"prepare.run_cpu_time_us":228,"prepare.run_wall_time_us":227}],["txn",{"num_ops":1,"prepare.queue_time_us":10996,"prepare.run_cpu_time_us":233,"prepare.run_wall_time_us":233}],["txn",{"num_ops":1,"prepare.queue_time_us":11089,"prepare.run_cpu_time_us":232,"prepare.run_wall_time_us":233}],["txn",{"num_ops":1,"prepare.queue_time_us":10976,"prepare.run_cpu_time_us":235,"prepare.run_wall_time_us":235}],["txn",{"num_ops":1,"prepare.queue_time_us":11041,"prepare.run_cpu_time_us":273,"prepare.run_wall_time_us":278}],["txn",{"num_ops":1,"prepare.queue_time_us":11180,"prepare.run_cpu_time_us":258,"prepare.run_wall_time_us":257}],["txn",{"num_ops":1,"prepare.queue_time_us":11304,"prepare.run_cpu_time_us":250,"prepare.run_wall_time_us":250}],["txn",{"num_ops":1,"prepare.queue_time_us":11132,"prepare.run_cpu_time_us":348,"prepare.run_wall_time_us":348}],["txn",{"num_ops":1,"prepare.queue_time_us":11311,"prepare.run_cpu_time_us":273,"prepare.run_wall_time_us":274}],["txn",{"num_ops":1,"prepare.queue_time_us":11450,"prepare.run_cpu_time_us":239,"prepare.run_wall_time_us":239}],["txn",{"num_ops":1,"prepare.queue_time_us":11545,"prepare.run_cpu_time_us":267,"prepare.run_wall_time_us":266}],["txn",{"num_ops":1,"prepare.queue_time_us":11686,"prepare.run_cpu_time_us":273,"prepare.run_wall_time_us":273}],["txn",{"num_ops":1,"prepare.queue_time_us":11829,"prepare.run_cpu_time_us":246,"prepare.run_wall_time_us":245}],["txn",{"mutex_wait_us":40,"num_ops":1,"prepare.queue_time_us":11860,"prepare.run_cpu_time_us":239,"prepare.run_wall_time_us":239}],["txn",{"num_ops":1,"prepare.queue_time_us":11873,"prepare.run_cpu_time_us":235,"prepare.run_wall_time_us":236}],["txn",{"num_ops":1,"prepare.queue_time_us":11984,"prepare.run_cpu_time_us":258,"prepare.run_wall_time_us":257}],["txn",{"num_ops":1,"prepare.queue_time_us":12113,"prepare.run_cpu_time_us":248,"prepare.run_wall_time_us":248}],["txn",{"num_ops":1,"prepare.queue_time_us":12214,"prepare.run_cpu_time_us":227,"prepare.run_wall_time_us":229}],["txn",{"num_ops":1,"prepare.queue_time_us":12293,"prepare.run_cpu_time_us":243,"prepare.run_wall_time_us":244}],["txn",{"mutex_wait_us":17,"num_ops":1,"prepare.queue_time_us":12397,"prepare.run_cpu_time_us":241,"prepare.run_wall_time_us":241}],["txn",{"num_ops":1,"prepare.queue_time_us":12477,"prepare.run_cpu_time_us":240,"prepare.run_wall_time_us":240}],["txn",{"num_ops":1,"prepare.queue_time_us":12585,"prepare.run_cpu_time_us":233,"prepare.run_wall_time_us":233}],["txn",{"num_ops":1,"prepare.queue_time_us":12652,"prepare.run_cpu_time_us":246,"prepare.run_wall_time_us":247}],["txn",{"num_ops":1,"prepare.queue_time_us":12750,"prepare.run_cpu_time_us":232,"prepare.run_wall_time_us":231}],["txn",{"num_ops":1,"prepare.queue_time_us":12846,"prepare.run_cpu_time_us":240,"prepare.run_wall_time_us":240}],["txn",{"num_ops":1,"prepare.queue_time_us":12958,"prepare.run_cpu_time_us":244,"prepare.run_wall_time_us":244}],["txn",{"num_ops":1,"prepare.queue_time_us":13052,"prepare.run_cpu_time_us":250,"prepare.run_wall_time_us":254}],["txn",{"num_ops":1,"prepare.queue_time_us":13120,"prepare.run_cpu_time_us":229,"prepare.run_wall_time_us":228}],["txn",{"num_ops":1,"prepare.queue_time_us":13207,"prepare.run_cpu_time_us":243,"prepare.run_wall_time_us":243}],["txn",{"mutex_wait_us":15,"num_ops":1,"prepare.queue_time_us":13312,"prepare.run_cpu_time_us":266,"prepare.run_wall_time_us":266}],["txn",{"num_ops":1,"prepare.queue_time_us":13201,"prepare.run_cpu_time_us":240,"prepare.run_wall_time_us":239}],["txn",{"num_ops":1,"prepare.queue_time_us":13294,"prepare.run_cpu_time_us":235,"prepare.run_wall_time_us":235}],["txn",{"num_ops":1,"prepare.queue_time_us":13397,"prepare.run_cpu_time_us":246,"prepare.run_wall_time_us":246}],["txn",{"num_ops":1,"prepare.queue_time_us":13511,"prepare.run_cpu_time_us":242,"prepare.run_wall_time_us":247}],["txn",{"num_ops":1,"prepare.queue_time_us":13620,"prepare.run_cpu_time_us":232,"prepare.run_wall_time_us":232}],["txn",{"num_ops":1,"prepare.queue_time_us":13724,"prepare.run_cpu_time_us":252,"prepare.run_wall_time_us":252}],["txn",{"num_ops":1,"prepare.queue_time_us":13855,"prepare.run_cpu_time_us":241,"prepare.run_wall_time_us":241}],["txn",{"num_ops":1,"prepare.queue_time_us":13958,"prepare.run_cpu_time_us":230,"prepare.run_wall_time_us":230}],["txn",{"num_ops":1,"prepare.queue_time_us":14045,"prepare.run_cpu_time_us":226,"prepare.run_wall_time_us":227}],["txn",{"num_ops":1,"prepare.queue_time_us":14137,"prepare.run_cpu_time_us":246,"prepare.run_wall_time_us":246}],["txn",{"num_ops":1,"prepare.queue_time_us":14225,"prepare.run_cpu_time_us":240,"prepare.run_wall_time_us":240}],["txn",{"num_ops":1,"prepare.queue_time_us":14321,"prepare.run_cpu_time_us":257,"prepare.run_wall_time_us":257}],["txn",{"num_ops":1,"prepare.queue_time_us":14450,"prepare.run_cpu_time_us":266,"prepare.run_wall_time_us":266}],["txn",{"num_ops":1,"prepare.queue_time_us":14575,"prepare.run_cpu_time_us":231,"prepare.run_wall_time_us":231}],["txn",{"num_ops":1,"prepare.queue_time_us":14659,"prepare.run_cpu_time_us":225,"prepare.run_wall_time_us":225}],["txn",{"num_ops":1,"prepare.queue_time_us":14748,"prepare.run_cpu_time_us":227,"prepare.run_wall_time_us":227}],["txn",{"num_ops":1,"prepare.queue_time_us":14819,"prepare.run_cpu_time_us":247,"prepare.run_wall_time_us":247}],["txn",{"num_ops":1,"prepare.queue_time_us":14934,"prepare.run_cpu_time_us":231,"prepare.run_wall_time_us":231}],["txn",{"num_ops":1,"prepare.queue_time_us":15033,"prepare.run_cpu_time_us":224,"prepare.run_wall_time_us":224}],["txn",{"num_ops":1,"prepare.queue_time_us":15124,"prepare.run_cpu_time_us":281,"prepare.run_wall_time_us":281}],["txn",{"num_ops":1,"prepare.queue_time_us":15270,"prepare.run_cpu_time_us":250,"prepare.run_wall_time_us":250}],["txn",{"num_ops":1,"prepare.queue_time_us":15390,"prepare.run_cpu_time_us":224,"prepare.run_wall_time_us":224}],["txn",{"num_ops":1,"prepare.queue_time_us":15480,"prepare.run_cpu_time_us":244,"prepare.run_wall_time_us":248}],["txn",{"num_ops":1,"prepare.queue_time_us":15600,"prepare.run_cpu_time_us":230,"prepare.run_wall_time_us":230}],["txn",{"num_ops":1,"prepare.queue_time_us":15700,"prepare.run_cpu_time_us":240,"prepare.run_wall_time_us":240}],["txn",{"num_ops":1,"prepare.queue_time_us":15794,"prepare.run_cpu_time_us":235,"prepare.run_wall_time_us":234}],["txn",{"num_ops":1,"prepare.queue_time_us":15886,"prepare.run_cpu_time_us":232,"prepare.run_wall_time_us":232}],["txn",{"num_ops":1,"prepare.queue_time_us":15985,"prepare.run_cpu_time_us":237,"prepare.run_wall_time_us":236}],["txn",{"num_ops":1,"prepare.queue_time_us":16082,"prepare.run_cpu_time_us":229,"prepare.run_wall_time_us":229}],["txn",{"mutex_wait_us":15,"num_ops":1,"prepare.queue_time_us":16159,"prepare.run_cpu_time_us":230,"prepare.run_wall_time_us":229}],["txn",{"num_ops":1,"prepare.queue_time_us":15973,"prepare.run_cpu_time_us":230,"prepare.run_wall_time_us":229}],["txn",{"num_ops":1,"prepare.queue_time_us":16044,"prepare.run_cpu_time_us":241,"prepare.run_wall_time_us":241}],["txn",{"num_ops":1,"prepare.queue_time_us":16156,"prepare.run_cpu_time_us":230,"prepare.run_wall_time_us":229}],["txn",{"num_ops":1,"prepare.queue_time_us":16239,"prepare.run_cpu_time_us":234,"prepare.run_wall_time_us":233}],["txn",{"num_ops":1,"prepare.queue_time_us":16289,"prepare.run_cpu_time_us":236,"prepare.run_wall_time_us":236}],["txn",{"num_ops":1,"prepare.queue_time_us":16387,"prepare.run_cpu_time_us":226,"prepare.run_wall_time_us":225}],["txn",{"num_ops":1,"prepare.queue_time_us":16410,"prepare.run_cpu_time_us":228,"prepare.run_wall_time_us":229}],["txn",{"num_ops":1,"prepare.queue_time_us":16493,"prepare.run_cpu_time_us":227,"prepare.run_wall_time_us":228}],["txn",{"num_ops":1,"prepare.queue_time_us":16584,"prepare.run_cpu_time_us":229,"prepare.run_wall_time_us":229}],["txn",{"num_ops":1,"prepare.queue_time_us":16637,"prepare.run_cpu_time_us":454,"prepare.run_wall_time_us":455}],["txn",{"num_ops":1,"prepare.queue_time_us":16930,"prepare.run_cpu_time_us":229,"prepare.run_wall_time_us":229}],["txn",{"num_ops":1,"prepare.queue_time_us":17027,"prepare.run_cpu_time_us":255,"prepare.run_wall_time_us":255}],["txn",{"mutex_wait_us":18,"num_ops":1,"prepare.queue_time_us":17147,"prepare.run_cpu_time_us":242,"prepare.run_wall_time_us":241}],["txn",{"num_ops":1,"prepare.queue_time_us":17266,"prepare.run_cpu_time_us":235,"prepare.run_wall_time_us":234}],["txn",{"mutex_wait_us":24,"num_ops":1,"prepare.queue_time_us":17354,"prepare.run_cpu_time_us":232,"prepare.run_wall_time_us":231}],["txn",{"num_ops":1,"prepare.queue_time_us":17417,"prepare.run_cpu_time_us":236,"prepare.run_wall_time_us":235}],["txn",{"num_ops":1,"prepare.queue_time_us":17409,"prepare.run_cpu_time_us":236,"prepare.run_wall_time_us":236}],["txn",{"num_ops":1,"prepare.queue_time_us":17449,"prepare.run_cpu_time_us":247,"prepare.run_wall_time_us":248}],["txn",{"num_ops":1,"prepare.queue_time_us":17551,"prepare.run_cpu_time_us":226,"prepare.run_wall_time_us":226}],["txn",{"num_ops":1,"prepare.queue_time_us":17638,"prepare.run_cpu_time_us":244,"prepare.run_wall_time_us":244}],["txn",{"num_ops":1,"prepare.queue_time_us":17749,"prepare.run_cpu_time_us":232,"prepare.run_wall_time_us":232}],["txn",{"num_ops":1,"prepare.queue_time_us":17842,"prepare.run_cpu_time_us":256,"prepare.run_wall_time_us":255}],["txn",{"num_ops":1,"prepare.queue_time_us":17956,"prepare.run_cpu_time_us":233,"prepare.run_wall_time_us":233}],["txn",{"num_ops":1,"prepare.queue_time_us":18058,"prepare.run_cpu_time_us":230,"prepare.run_wall_time_us":229}],["txn",{"mutex_wait_us":4,"num_ops":1,"prepare.queue_time_us":18148,"prepare.run_cpu_time_us":227,"prepare.run_wall_time_us":227}],["txn",{"num_ops":1,"prepare.queue_time_us":18224,"prepare.run_cpu_time_us":236,"prepare.run_wall_time_us":235}],["txn",{"num_ops":1,"prepare.queue_time_us":18293,"prepare.run_cpu_time_us":233,"prepare.run_wall_time_us":233}],["txn",{"num_ops":1,"prepare.queue_time_us":18378,"prepare.run_cpu_time_us":244,"prepare.run_wall_time_us":249}],["txn",{"num_ops":1,"prepare.queue_time_us":18491,"prepare.run_cpu_time_us":251,"prepare.run_wall_time_us":251}],["txn",{"num_ops":1,"prepare.queue_time_us":18600,"prepare.run_cpu_time_us":230,"prepare.run_wall_time_us":231}],["txn",{"num_ops":1,"prepare.queue_time_us":18675,"prepare.run_cpu_time_us":227,"prepare.run_wall_time_us":227}],["txn",{"num_ops":1,"prepare.queue_time_us":18751,"prepare.run_cpu_time_us":250,"prepare.run_wall_time_us":251}],["txn",{"num_ops":1,"prepare.queue_time_us":18787,"prepare.run_cpu_time_us":263,"prepare.run_wall_time_us":262}],["txn",{"num_ops":1,"prepare.queue_time_us":18914,"prepare.run_cpu_time_us":232,"prepare.run_wall_time_us":232}],["txn",{"num_ops":1,"prepare.queue_time_us":18776,"prepare.run_cpu_time_us":239,"prepare.run_wall_time_us":238}],["txn",{"num_ops":1,"prepare.queue_time_us":18815,"prepare.run_cpu_time_us":237,"prepare.run_wall_time_us":236}],["txn",{"num_ops":1,"prepare.queue_time_us":18890,"prepare.run_cpu_time_us":235,"prepare.run_wall_time_us":235}],["txn",{"num_ops":1,"prepare.queue_time_us":18984,"prepare.run_cpu_time_us":224,"prepare.run_wall_time_us":225}],["txn",{"num_ops":1,"prepare.queue_time_us":19064,"prepare.run_cpu_time_us":231,"prepare.run_wall_time_us":231}],["txn",{"num_ops":1,"prepare.queue_time_us":19166,"prepare.run_cpu_time_us":231,"prepare.run_wall_time_us":231}],["txn",{"num_ops":1,"prepare.queue_time_us":19267,"prepare.run_cpu_time_us":229,"prepare.run_wall_time_us":228}],["txn",{"num_ops":1,"prepare.queue_time_us":19365,"prepare.run_cpu_time_us":232,"prepare.run_wall_time_us":236}],["txn",{"mutex_wait_us":17,"num_ops":1,"prepare.queue_time_us":19446,"prepare.run_cpu_time_us":238,"prepare.run_wall_time_us":238}],["txn",{"num_ops":1,"prepare.queue_time_us":19572,"prepare.run_cpu_time_us":292,"prepare.run_wall_time_us":292}],["txn",{"num_ops":1,"prepare.queue_time_us":19696,"prepare.run_cpu_time_us":259,"prepare.run_wall_time_us":258}],["txn",{"num_ops":1,"prepare.queue_time_us":19801,"prepare.run_cpu_time_us":272,"prepare.run_wall_time_us":272,"spinlock_wait_cycles":1920}],["txn",{"num_ops":1,"prepare.queue_time_us":19934,"prepare.run_cpu_time_us":232,"prepare.run_wall_time_us":232}],["txn",{"num_ops":1,"prepare.queue_time_us":20030,"prepare.run_cpu_time_us":228,"prepare.run_wall_time_us":229}],["txn",{"num_ops":1,"prepare.queue_time_us":20103,"prepare.run_cpu_time_us":236,"prepare.run_wall_time_us":236}],["txn",{"num_ops":1,"prepare.queue_time_us":20209,"prepare.run_cpu_time_us":243,"prepare.run_wall_time_us":243}],["txn",{"num_ops":1,"prepare.queue_time_us":20319,"prepare.run_cpu_time_us":237,"prepare.run_wall_time_us":236}],["txn",{"num_ops":1,"prepare.queue_time_us":20424,"prepare.run_cpu_time_us":226,"prepare.run_wall_time_us":225}],["txn",{"num_ops":1,"prepare.queue_time_us":20497,"prepare.run_cpu_time_us":231,"prepare.run_wall_time_us":231}],["txn",{"num_ops":1,"prepare.queue_time_us":20534,"prepare.run_cpu_time_us":227,"prepare.run_wall_time_us":227}],["txn",{"num_ops":1,"prepare.queue_time_us":20602,"prepare.run_cpu_time_us":227,"prepare.run_wall_time_us":227}],["txn",{"num_ops":1,"prepare.queue_time_us":20692,"prepare.run_cpu_time_us":228,"prepare.run_wall_time_us":228}],["txn",{"num_ops":1,"prepare.queue_time_us":20788,"prepare.run_cpu_time_us":243,"prepare.run_wall_time_us":242}],["txn",{"num_ops":1,"prepare.queue_time_us":20894,"prepare.run_cpu_time_us":228,"prepare.run_wall_time_us":228}],["txn",{"num_ops":1,"prepare.queue_time_us":20984,"prepare.run_cpu_time_us":230,"prepare.run_wall_time_us":230}],["txn",{"num_ops":1,"prepare.queue_time_us":21069,"prepare.run_cpu_time_us":247,"prepare.run_wall_time_us":246}],["txn",{"num_ops":1,"prepare.queue_time_us":21184,"prepare.run_cpu_time_us":244,"prepare.run_wall_time_us":243}],["txn",{"num_ops":1,"prepare.queue_time_us":21084,"prepare.run_cpu_time_us":251,"prepare.run_wall_time_us":251}],["txn",{"num_ops":1,"prepare.queue_time_us":21190,"prepare.run_cpu_time_us":245,"prepare.run_wall_time_us":245}],["txn",{"num_ops":1,"prepare.queue_time_us":21264,"prepare.run_cpu_time_us":242,"prepare.run_wall_time_us":242}],["txn",{"num_ops":1,"prepare.queue_time_us":21357,"prepare.run_cpu_time_us":261,"prepare.run_wall_time_us":261}],["txn",{"num_ops":1,"prepare.queue_time_us":21467,"prepare.run_cpu_time_us":237,"prepare.run_wall_time_us":237}],["txn",{"mutex_wait_us":14,"num_ops":1,"prepare.queue_time_us":21572,"prepare.run_cpu_time_us":250,"prepare.run_wall_time_us":249}],["txn",{"num_ops":1,"prepare.queue_time_us":21669,"prepare.run_cpu_time_us":236,"prepare.run_wall_time_us":235}],["txn",{"num_ops":1,"prepare.queue_time_us":21762,"prepare.run_cpu_time_us":449,"prepare.run_wall_time_us":448}],["txn",{"num_ops":1,"prepare.queue_time_us":22072,"prepare.run_cpu_time_us":260,"prepare.run_wall_time_us":260}],["txn",{"num_ops":1,"prepare.queue_time_us":22198,"prepare.run_cpu_time_us":244,"prepare.run_wall_time_us":244}],["txn",{"num_ops":1,"prepare.queue_time_us":22306,"prepare.run_cpu_time_us":233,"prepare.run_wall_time_us":233}],["txn",{"num_ops":1,"prepare.queue_time_us":22391,"prepare.run_cpu_time_us":249,"prepare.run_wall_time_us":248}],["txn",{"num_ops":1,"prepare.queue_time_us":22498,"prepare.run_cpu_time_us":230,"prepare.run_wall_time_us":231}],["txn",{"num_ops":1,"prepare.queue_time_us":22590,"prepare.run_cpu_time_us":243,"prepare.run_wall_time_us":243}],["txn",{"num_ops":1,"prepare.queue_time_us":22118,"prepare.run_cpu_time_us":441,"prepare.run_wall_time_us":445}],["txn",{"num_ops":1,"prepare.queue_time_us":22389,"prepare.run_cpu_time_us":260,"prepare.run_wall_time_us":262}],["txn",{"num_ops":1,"prepare.queue_time_us":22505,"prepare.run_cpu_time_us":241,"prepare.run_wall_time_us":242}],["txn",{"num_ops":1,"prepare.queue_time_us":22609,"prepare.run_cpu_tim I0819 06:34:28.964249 1150 raft_consensus-itest.cc:164] Waiting for row count 0... I0819 06:34:29.288964 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:29.289130 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:29.289238 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:29.289371 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 31639 bytes I0819 06:34:29.294472 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 227 or (T in {228,229,231,232,233,234,235,236,237,238,239})}] I0819 06:34:29.302930 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:29.359416 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:29.366847 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:29.367107 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 304 or (T in {305,306,307,308,309,304})}] I0819 06:34:29.405493 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 222 rows (9403 bytes) I0819 06:34:29.406168 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.131s user 0.064s sys 0.024s I0819 06:34:29.406359 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":2570,"data dir 0.run_cpu_time_us":507,"data dir 0.run_wall_time_us":1697,"lbm_read_time_us":191,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2256,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":19,"spinlock_wait_cycles":10665856,"thread_start_us":1757,"threads_started":1} I0819 06:34:29.547762 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:29.547909 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:29.548019 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:29.548113 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 31639 bytes I0819 06:34:29.548265 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 455}] I0819 06:34:29.549479 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:29.594899 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:29.618787 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:29.619036 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 517}] I0819 06:34:29.645511 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 231 rows (9706 bytes) I0819 06:34:29.646106 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.105s user 0.060s sys 0.004s I0819 06:34:29.646280 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":102,"data dir 0.run_cpu_time_us":476,"data dir 0.run_wall_time_us":2867,"lbm_read_time_us":181,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1010,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":33789568} I0819 06:34:29.771373 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:29.771639 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:29.771780 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:29.771914 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 31639 bytes I0819 06:34:29.772918 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 636}] I0819 06:34:29.775526 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:29.826951 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:29.835089 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:29.835342 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 726 or (T in {727,729,730,731,732,733,734,735,728,726})}] I0819 06:34:29.864706 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 180 rows (8620 bytes) I0819 06:34:29.865365 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.100s user 0.060s sys 0.000s I0819 06:34:29.865569 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":106,"data dir 0.run_cpu_time_us":502,"data dir 0.run_wall_time_us":3116,"lbm_read_time_us":235,"lbm_reads_lt_1ms":4,"lbm_write_time_us":966,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":10807552} I0819 06:34:30.005769 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:30.006054 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:30.006191 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:30.006325 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 31639 bytes I0819 06:34:30.006495 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 913}] I0819 06:34:30.008085 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:30.094771 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:30.120470 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:30.120741 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 1038 or (T in {1039,1040,1038})}] I0819 06:34:30.167690 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 278 rows (10746 bytes) I0819 06:34:30.168933 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.175s user 0.080s sys 0.004s I0819 06:34:30.169140 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":114,"data dir 0.run_cpu_time_us":381,"data dir 0.run_wall_time_us":2860,"lbm_read_time_us":174,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1238,"lbm_writes_lt_1ms":20} I0819 06:34:30.327378 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:30.327553 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:30.327662 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:30.327764 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 63511 bytes I0819 06:34:30.327906 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 1232 or (T in {1233,1235})}] I0819 06:34:30.329452 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:30.381763 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:30.393949 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:30.394192 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 1287 or (T in {1288,1290,1289,1291,1293,1292,1295,1294,1296,1298,1297,1299,1300,1301,1303,1304,1305,1306,1307,1302,1287})}] I0819 06:34:30.424655 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 311 rows (11650 bytes) I0819 06:34:30.425272 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.107s user 0.076s sys 0.000s I0819 06:34:30.425390 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":1148,"data dir 0.run_cpu_time_us":363,"data dir 0.run_wall_time_us":2481,"lbm_read_time_us":159,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1139,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":13743872} I0819 06:34:30.554560 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:30.554831 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:30.554978 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:30.555120 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 31639 bytes I0819 06:34:30.555304 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 1453}] I0819 06:34:30.557150 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:30.594200 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:30.601151 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:30.601399 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 1489 or (T in {1491,1492,1493,1494,1490,1496,1495,1497,1489})}] I0819 06:34:30.634461 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 229 rows (9908 bytes) I0819 06:34:30.635723 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.087s user 0.068s sys 0.004s I0819 06:34:30.635928 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":351,"data dir 0.run_cpu_time_us":358,"data dir 0.run_wall_time_us":2449,"lbm_read_time_us":190,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1149,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":10036480} I0819 06:34:30.750851 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:30.750990 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:30.751092 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:30.751178 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 31639 bytes I0819 06:34:30.751303 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 1630 or (T in {1632})}] I0819 06:34:30.752576 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:30.779379 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:30.783702 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:30.783836 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 1655 or (T in {1656,1657,1658,1659,1660,1661,1662,1663,1664,1665,1666,1667,1668,1669,1670,1671,1672,1674,1673,1675,1676,1677,1678,1679,1680,1681,1682,1683,1684,1685,1687,1688,1686,1689,1690,1692,1693,1691,1655})}] I0819 06:34:30.807205 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 177 rows (8748 bytes) I0819 06:34:30.807698 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.066s user 0.056s sys 0.000s I0819 06:34:30.807883 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":923,"data dir 0.run_cpu_time_us":346,"data dir 0.run_wall_time_us":2505,"lbm_read_time_us":164,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1016,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":5036288} I0819 06:34:30.915449 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:30.915626 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:30.915738 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:30.915848 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 31639 bytes I0819 06:34:30.916002 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 1872 or (T in {1873,1874})}] I0819 06:34:30.917418 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:30.947985 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:30.952823 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:30.953059 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 1925 or (T in {1927,1928,1929,1930,1931,1926,1925})}] I0819 06:34:30.977018 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 242 rows (10222 bytes) I0819 06:34:30.977519 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.066s user 0.056s sys 0.000s I0819 06:34:30.977630 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":1334,"data dir 0.run_cpu_time_us":348,"data dir 0.run_wall_time_us":2594,"lbm_read_time_us":165,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1082,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":2337920} I0819 06:34:31.061511 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:31.061689 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:31.061800 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:31.061904 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 31639 bytes I0819 06:34:31.062050 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 2044}] I0819 06:34:31.063540 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:31.089992 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:31.094782 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:31.096637 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 2103}] I0819 06:34:31.112705 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 172 rows (8623 bytes) I0819 06:34:31.113116 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.057s user 0.040s sys 0.004s I0819 06:34:31.113225 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":1292,"data dir 0.run_cpu_time_us":363,"data dir 0.run_wall_time_us":2615,"lbm_read_time_us":127,"lbm_reads_lt_1ms":4,"lbm_write_time_us":913,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":1848832} W0819 06:34:31.161975 2936 raft_consensus.cc:1244] Rejecting consensus request: Soft memory limit exceeded (at 185.65% of capacity) I0819 06:34:31.162492 2936 rpcz_store.cc:249] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49244 (request call id 7) took 2137ms. Request Metrics: {} I0819 06:34:31.163941 1150 raft_consensus-itest.cc:164] Waiting for row count 10000... I0819 06:34:31.182507 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:31.182680 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:31.182788 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:31.182894 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 31639 bytes I0819 06:34:31.184181 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 2235}] I0819 06:34:31.185616 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:31.212378 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:31.216284 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:31.216539 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 2261}] I0819 06:34:31.240030 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 190 rows (9038 bytes) I0819 06:34:31.240710 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.061s user 0.044s sys 0.000s I0819 06:34:31.240922 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":1381,"data dir 0.run_cpu_time_us":327,"data dir 0.run_wall_time_us":2530,"lbm_read_time_us":163,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3477,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":19,"spinlock_wait_cycles":9383680} I0819 06:34:31.310037 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:31.310201 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:31.310304 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:31.310410 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 15703 bytes I0819 06:34:31.310556 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 2369}] I0819 06:34:31.311981 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:31.334699 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:31.338773 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:31.339021 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 2413 or (T in {2415,2416,2414,2413})}] I0819 06:34:31.360071 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 135 rows (7780 bytes) I0819 06:34:31.360733 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.053s user 0.044s sys 0.000s I0819 06:34:31.360937 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":992,"data dir 0.run_cpu_time_us":369,"data dir 0.run_wall_time_us":2597,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1123,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":2485248} I0819 06:34:31.434823 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:31.435010 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:31.435123 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:31.435232 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 31639 bytes I0819 06:34:31.435380 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 2528 or (T in {2530})}] I0819 06:34:31.436777 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:31.493912 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:31.503615 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:31.503859 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 2614 or (T in {2614})}] I0819 06:34:31.525351 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 159 rows (8326 bytes) I0819 06:34:31.526001 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.095s user 0.048s sys 0.004s I0819 06:34:31.526182 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":1052,"data dir 0.run_cpu_time_us":391,"data dir 0.run_wall_time_us":2532,"lbm_read_time_us":160,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1121,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":4495616} I0819 06:34:31.598227 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:31.598398 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:31.598507 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:31.598604 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 31639 bytes I0819 06:34:31.598747 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 2724}] I0819 06:34:31.600200 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:31.627100 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:31.644819 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:31.644970 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 2759 or (T in {2760,2761,2762,2759})}] I0819 06:34:31.667742 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 196 rows (9163 bytes) I0819 06:34:31.668287 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.075s user 0.048s sys 0.000s I0819 06:34:31.668455 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":1337,"data dir 0.run_cpu_time_us":367,"data dir 0.run_wall_time_us":2612,"lbm_read_time_us":152,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1132,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":3924480} I0819 06:34:31.741196 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:31.741454 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:31.741581 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:31.741711 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 31639 bytes I0819 06:34:31.741876 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 2921}] I0819 06:34:31.743315 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:31.771986 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:31.775228 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:31.775364 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 2954}] I0819 06:34:31.796262 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 197 rows (9199 bytes) I0819 06:34:31.796934 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.060s user 0.052s sys 0.000s I0819 06:34:31.797123 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":1110,"data dir 0.run_cpu_time_us":332,"data dir 0.run_wall_time_us":2434,"lbm_read_time_us":183,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1094,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":3528320} I0819 06:34:31.863423 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:31.863597 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:31.863706 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:31.863816 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 31639 bytes I0819 06:34:31.863960 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 3062 or (T in {3063})}] I0819 06:34:31.865392 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:31.888686 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:31.891746 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:31.891882 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 3100}] I0819 06:34:31.911556 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 141 rows (7922 bytes) I0819 06:34:31.912098 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.052s user 0.044s sys 0.000s I0819 06:34:31.912274 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":1023,"data dir 0.run_cpu_time_us":344,"data dir 0.run_wall_time_us":2368,"lbm_read_time_us":189,"lbm_reads_lt_1ms":4,"lbm_write_time_us":895,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":2829312} I0819 06:34:31.982342 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:31.982517 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:31.982625 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:31.982718 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 15703 bytes I0819 06:34:31.982873 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 3200 or (T in {3201})}] I0819 06:34:31.984267 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:32.036893 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:32.040316 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:32.043964 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 3281}] I0819 06:34:32.069779 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 136 rows (7773 bytes) I0819 06:34:32.070410 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.094s user 0.044s sys 0.000s I0819 06:34:32.070590 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":986,"data dir 0.run_cpu_time_us":326,"data dir 0.run_wall_time_us":2328,"lbm_read_time_us":140,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1073,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":3964032} I0819 06:34:32.142709 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:32.142906 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:32.143018 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:32.143127 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 31639 bytes I0819 06:34:32.143275 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 3360 or (T in {3362})}] I0819 06:34:32.144747 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:32.172888 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:32.180101 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:32.180450 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 3418}] I0819 06:34:32.202904 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 162 rows (8402 bytes) I0819 06:34:32.203532 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.064s user 0.052s sys 0.000s I0819 06:34:32.203723 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":1007,"data dir 0.run_cpu_time_us":340,"data dir 0.run_wall_time_us":2354,"lbm_read_time_us":167,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1218,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":12135296} I0819 06:34:32.274463 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:32.274704 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:32.274833 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:32.274968 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 31639 bytes I0819 06:34:32.275130 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 3559 or (T in {3560})}] I0819 06:34:32.276548 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:32.306985 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:32.312508 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:32.312743 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 3650 or (T in {3651,3652,3650})}] I0819 06:34:32.353869 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 199 rows (9230 bytes) I0819 06:34:32.354461 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.084s user 0.056s sys 0.000s I0819 06:34:32.354652 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":1093,"data dir 0.run_cpu_time_us":351,"data dir 0.run_wall_time_us":2634,"lbm_read_time_us":165,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1071,"lbm_writes_lt_1ms":20,"mutex_wait_us":14390,"spinlock_wait_cycles":2519040} I0819 06:34:32.425462 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:32.425634 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:32.425745 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:32.425855 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 31639 bytes I0819 06:34:32.427038 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 3828}] I0819 06:34:32.429410 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:32.482525 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:32.498706 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:32.499994 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 3899 or (T in {3900,3901,3902,3903,3904,3905,3906,3907,3899})}] I0819 06:34:32.539563 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 267 rows (10786 bytes) I0819 06:34:32.540210 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.118s user 0.072s sys 0.000s I0819 06:34:32.540449 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":1078,"data dir 0.run_cpu_time_us":311,"data dir 0.run_wall_time_us":2445,"lbm_read_time_us":205,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1128,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":6994048} I0819 06:34:32.606328 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:32.606508 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:32.606621 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:32.606951 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 31639 bytes I0819 06:34:32.607720 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 4054}] I0819 06:34:32.609195 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:32.643486 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:32.658071 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:32.658212 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 4108 or (T in {4108})}] I0819 06:34:32.682101 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 227 rows (9880 bytes) I0819 06:34:32.682713 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.080s user 0.060s sys 0.000s I0819 06:34:32.682914 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":967,"data dir 0.run_cpu_time_us":329,"data dir 0.run_wall_time_us":2318,"lbm_read_time_us":122,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1129,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":2002688} I0819 06:34:32.748739 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:32.748934 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:32.749045 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:32.749155 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 31639 bytes I0819 06:34:32.749295 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 4277}] I0819 06:34:32.750680 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:32.789327 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:32.793473 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:32.793723 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 4345}] I0819 06:34:32.819691 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 223 rows (9783 bytes) I0819 06:34:32.820335 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.075s user 0.064s sys 0.000s I0819 06:34:32.820549 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":992,"data dir 0.run_cpu_time_us":320,"data dir 0.run_wall_time_us":2461,"lbm_read_time_us":169,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1077,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":8739584} I0819 06:34:32.885424 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:32.885677 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:32.885823 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:32.885962 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 31639 bytes I0819 06:34:32.886901 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 4483 or (T in {4484})}] I0819 06:34:32.888362 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:32.930029 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:32.935376 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:32.935612 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 4565 or (T in {4566,4567,4568,4569,4570,4571,4572,4573,4574,4575,4576,4577,4578,4565})}] I0819 06:34:32.964742 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 206 rows (9397 bytes) I0819 06:34:32.965912 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.085s user 0.060s sys 0.004s I0819 06:34:32.968904 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":1006,"data dir 0.run_cpu_time_us":332,"data dir 0.run_wall_time_us":2461,"lbm_read_time_us":180,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1124,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":3635456} I0819 06:34:33.091795 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:33.092037 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:33.092169 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:33.092301 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 63511 bytes I0819 06:34:33.092499 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 4824 or (T in {4826,4827})}] I0819 06:34:33.093969 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:33.142208 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:33.150285 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:33.150423 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 4871 or (T in {4872,4873,4874,4875,4876,4877,4878,4879,4880,4881,4882,4883,4884,4885,4886,4887,4888,4889,4891,4892,4893,4894,4895,4896,4897,4898,4899,4900,4901,4890,4871})}] I0819 06:34:33.188716 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 342 rows (12463 bytes) I0819 06:34:33.189613 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.103s user 0.080s sys 0.000s I0819 06:34:33.189728 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":1009,"data dir 0.run_cpu_time_us":309,"data dir 0.run_wall_time_us":2430,"lbm_read_time_us":158,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1164,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":15232} I0819 06:34:33.274400 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:33.274576 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:33.274688 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:33.274799 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 63511 bytes I0819 06:34:33.274947 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 5135 or (T in {5136,5137,5138,5139,5140,5141,5142,5143,5144,5145,5146,5147,5148,5149,5150,5151,5152,5153,5154,5155,5156,5158,5157})}] I0819 06:34:33.276386 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:33.316802 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:33.319631 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:33.319856 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 5253}] I0819 06:34:33.349179 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 281 rows (11086 bytes) I0819 06:34:33.351691 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.100s user 0.068s sys 0.004s I0819 06:34:33.351897 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":1136,"data dir 0.run_cpu_time_us":356,"data dir 0.run_wall_time_us":2539,"lbm_read_time_us":161,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1045,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":2968576} I0819 06:34:33.412714 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:33.412950 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:33.413081 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:33.413211 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 63511 bytes I0819 06:34:33.413399 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 5398}] I0819 06:34:33.414831 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:33.456446 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:33.462960 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:33.463217 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 5444 or (T in {5445,5446,5448,5450,5447,5451,5452,5453,5454,5455,5456,5457,5458,5459,5460,5449,5444})}] I0819 06:34:33.498736 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 293 rows (11383 bytes) I0819 06:34:33.499338 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.090s user 0.072s sys 0.000s I0819 06:34:33.499528 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":1123,"data dir 0.run_cpu_time_us":333,"data dir 0.run_wall_time_us":2472,"lbm_read_time_us":167,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1145,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":986112} I0819 06:34:33.554211 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:33.554397 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:33.554509 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:33.554620 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 31639 bytes I0819 06:34:33.554764 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 5563}] I0819 06:34:33.556162 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:33.583329 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:33.585425 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:33.585630 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 5615}] I0819 06:34:33.606518 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 165 rows (8481 bytes) I0819 06:34:33.607131 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.055s user 0.052s sys 0.000s I0819 06:34:33.607317 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":1033,"data dir 0.run_cpu_time_us":345,"data dir 0.run_wall_time_us":2323,"lbm_read_time_us":163,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1102,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":3810944} I0819 06:34:33.671733 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:33.671916 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:33.672024 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:33.672123 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 31639 bytes I0819 06:34:33.672269 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 5727 or (T in {5728,5729,5730,5731,5732,5733,5734,5735,5736,5737,5738,5739,5740,5741,5742,5743,5744})}] I0819 06:34:33.673738 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:33.702702 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:33.707530 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:33.707711 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 5808}] I0819 06:34:33.740464 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 163 rows (8431 bytes) I0819 06:34:33.741653 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.080s user 0.052s sys 0.000s I0819 06:34:33.741885 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":1050,"data dir 0.run_cpu_time_us":365,"data dir 0.run_wall_time_us":2578,"lbm_read_time_us":171,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1122,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":5257472} I0819 06:34:33.802837 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:33.803021 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:33.803134 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:33.803259 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 31639 bytes I0819 06:34:33.803413 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 5957 or (T in {5959,5958})}] I0819 06:34:33.805075 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:33.839186 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:33.844802 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:33.845046 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6032 or (T in {6033,6035,6036,6037,6038,6039,6040,6041,6034,6032})}] I0819 06:34:33.883582 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 231 rows (9964 bytes) I0819 06:34:33.897258 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.097s user 0.064s sys 0.000s I0819 06:34:33.906388 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":1127,"data dir 0.run_cpu_time_us":306,"data dir 0.run_wall_time_us":2523,"lbm_read_time_us":203,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1286,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":1112704} I0819 06:34:33.958169 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:33.958303 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:33.958405 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:33.958508 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 63511 bytes I0819 06:34:33.958642 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6260 or (T in {6262,6261,6263,6264})}] I0819 06:34:33.959744 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:34.014487 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:34.022260 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:34.022511 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6348}] I0819 06:34:34.067225 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 307 rows (11697 bytes) I0819 06:34:34.069541 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.115s user 0.072s sys 0.000s I0819 06:34:34.070462 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":969,"data dir 0.run_cpu_time_us":321,"data dir 0.run_wall_time_us":2409,"lbm_read_time_us":164,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1024,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":2281216} I0819 06:34:34.161401 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:34.161581 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:34.161695 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:34.161804 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 63511 bytes I0819 06:34:34.161945 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6562}] I0819 06:34:34.164264 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:34.252416 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:34.255236 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:34.255477 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6788}] I0819 06:34:34.302188 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 298 rows (11506 bytes) I0819 06:34:34.302765 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.143s user 0.072s sys 0.000s I0819 06:34:34.315627 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":1014,"data dir 0.run_cpu_time_us":324,"data dir 0.run_wall_time_us":2392,"lbm_read_time_us":153,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1230,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":1047040} I0819 06:34:34.354755 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:34.354933 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:34.355043 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:34.355162 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 63511 bytes I0819 06:34:34.355312 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6972 or (T in {6974,6975})}] I0819 06:34:34.356775 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:34.409876 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:34.415000 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:34.415274 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 7087}] I0819 06:34:34.483459 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 410 rows (13978 bytes) I0819 06:34:34.484097 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.134s user 0.092s sys 0.000s I0819 06:34:34.484288 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":904,"data dir 0.run_cpu_time_us":311,"data dir 0.run_wall_time_us":2384,"lbm_read_time_us":262,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1196,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":5278080} I0819 06:34:34.573899 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:34.574046 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:34.574160 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:34.574246 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 63511 bytes I0819 06:34:34.574368 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 7386 or (T in {7387,7388,7389,7390,7391,7392,7393,7394,7395,7396,7398})}] I0819 06:34:34.575667 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:34.626109 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:34.640275 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:34.642207 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 7469 or (T in {7472,7473,7474,7475,7476,7471,7478,7479,7480,7481,7470,7477,7469})}] I0819 06:34:34.680353 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 397 rows (13670 bytes) I0819 06:34:34.682122 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.122s user 0.084s sys 0.000s I0819 06:34:34.682350 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":1056,"data dir 0.run_cpu_time_us":323,"data dir 0.run_wall_time_us":2505,"lbm_read_time_us":172,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1155,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":1757824} I0819 06:34:34.718101 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:34.718291 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:34.718412 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:34.718533 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 31639 bytes I0819 06:34:34.718847 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 7642 or (T in {7643})}] I0819 06:34:34.720326 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:34.761430 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:34.763852 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:34.764108 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 7732 or (T in {7734,7733})}] I0819 06:34:34.800868 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 273 rows (10925 bytes) I0819 06:34:34.802141 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.087s user 0.072s sys 0.000s I0819 06:34:34.802346 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":1335,"data dir 0.run_cpu_time_us":322,"data dir 0.run_wall_time_us":2474,"lbm_read_time_us":165,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1175,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":18189440} I0819 06:34:34.829931 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:34.830130 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:34.830246 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:34.830354 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 31639 bytes I0819 06:34:34.830503 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 7864}] I0819 06:34:34.831928 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:34.863622 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:34.872439 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:34.872622 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 7916 or (T in {7917,7918,7919,7920,7921,7923,7922,7916})}] I0819 06:34:34.902217 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 222 rows (9761 bytes) I0819 06:34:34.903365 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.077s user 0.056s sys 0.000s I0819 06:34:34.903578 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":961,"data dir 0.run_cpu_time_us":317,"data dir 0.run_wall_time_us":2332,"lbm_read_time_us":279,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1082,"lbm_writes_lt_1ms":20} I0819 06:34:34.928542 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:34.928731 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:34.928845 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:34.928954 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 31639 bytes I0819 06:34:34.929292 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 8068}] I0819 06:34:34.930409 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:34.956019 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:34.958330 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:34.958506 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 8107 or (T in {8108})}] I0819 06:34:34.985020 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 203 rows (9361 bytes) I0819 06:34:34.986171 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.060s user 0.052s sys 0.000s I0819 06:34:34.986337 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":1070,"data dir 0.run_cpu_time_us":323,"data dir 0.run_wall_time_us":2493,"lbm_read_time_us":170,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1012,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":382720} I0819 06:34:35.021392 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:35.021582 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:35.021704 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:35.021829 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 15703 bytes I0819 06:34:35.021987 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 8204}] I0819 06:34:35.023469 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:35.045325 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:35.047809 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:35.047978 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 8229}] I0819 06:34:35.072348 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 122 rows (7504 bytes) I0819 06:34:35.072952 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.063s user 0.044s sys 0.000s I0819 06:34:35.073107 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":924,"data dir 0.run_cpu_time_us":356,"data dir 0.run_wall_time_us":2342,"lbm_read_time_us":159,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1105,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":476544} I0819 06:34:35.102903 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:35.103093 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:35.103212 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:35.103327 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 15703 bytes I0819 06:34:35.103471 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 8267}] I0819 06:34:35.104903 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:35.122750 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:35.125349 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:35.125522 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 8294}] I0819 06:34:35.172520 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 78 rows (6480 bytes) I0819 06:34:35.173146 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.073s user 0.036s sys 0.000s I0819 06:34:35.173337 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":1029,"data dir 0.run_cpu_time_us":345,"data dir 0.run_wall_time_us":2483,"lbm_read_time_us":162,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1127,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":23296} I0819 06:34:35.203824 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:35.203996 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:35.204113 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:35.204226 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 31639 bytes I0819 06:34:35.204375 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 8462 or (T in {8463})}] I0819 06:34:35.205966 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:35.265349 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:35.267985 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:35.268193 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 8594}] I0819 06:34:35.313218 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 194 rows (9143 bytes) I0819 06:34:35.314927 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.114s user 0.060s sys 0.000s I0819 06:34:35.315147 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":1331,"data dir 0.run_cpu_time_us":347,"data dir 0.run_wall_time_us":2560,"lbm_read_time_us":171,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1106,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":169856} I0819 06:34:35.355947 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:35.356135 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:35.356238 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:35.356330 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 63511 bytes I0819 06:34:35.358247 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 8761}] I0819 06:34:35.359733 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:35.444434 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:35.446939 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:35.448554 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 9011 or (T in {9012,9013,9014,9015,9016,9017,9018,9019,9020,9021,9022,9023,9024,9025,9026,9027,9028})}] I0819 06:34:35.595628 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 294 rows (11411 bytes) I0819 06:34:35.596315 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.244s user 0.076s sys 0.000s I0819 06:34:35.596544 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":1003,"data dir 0.run_cpu_time_us":332,"data dir 0.run_wall_time_us":2580,"lbm_read_time_us":166,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1188,"lbm_writes_lt_1ms":20,"spinlock_wait_cycles":3793664} I0819 06:34:35.626900 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:35.627089 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:35.627202 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:35.627310 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 127255 bytes I0819 06:34:35.627452 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 9538 or (T in {9539})}] I0819 06:34:35.628916 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:35.733376 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:35.738199 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:35.738442 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 9707 or (T in {9708,9709,9711,9710,9707})}] I0819 06:34:35.845495 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 783 rows (22179 bytes) I0819 06:34:35.846958 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.222s user 0.156s sys 0.000s I0819 06:34:35.847087 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":1095,"data dir 0.run_cpu_time_us":349,"data dir 0.run_wall_time_us":2446,"lbm_read_time_us":170,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1219,"lbm_writes_lt_1ms":20} I0819 06:34:35.852411 2864 tablet.cc:1040] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:35.852548 2864 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:35.852643 2864 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:35.852728 2864 tablet.cc:1042] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Memstore in-memory size: 63511 bytes I0819 06:34:35.852846 2864 tablet.cc:1305] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 10001 or (T in {10001})}] I0819 06:34:35.853878 2864 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:35.889616 2864 tablet.cc:1390] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:35.891283 2864 tablet.cc:1445] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:35.891412 2864 tablet.cc:1447] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Phase 2 snapshot: MvccSnapshot[committed={T|T < 10001 or (T in {10001})}] I0819 06:34:35.911090 2864 tablet.cc:1489] T 31d2fe4307b848379aea3d36390148e4 P 4c94be5808b34fc3bb2d82e3988c20f1: Flush successful on 464 rows (15143 bytes) I0819 06:34:35.912183 2864 maintenance_manager.cc:466] Time spent running FlushMRSOp(31d2fe4307b848379aea3d36390148e4): real 0.061s user 0.060s sys 0.000s I0819 06:34:35.912312 2864 maintenance_manager.cc:472] P 4c94be5808b34fc3bb2d82e3988c20f1: FlushMRSOp(31d2fe4307b848379aea3d36390148e4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":1341,"data dir 0.run_cpu_time_us":399,"data dir 0.run_wall_time_us":2401,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4,"lbm_write_time_us":765,"lbm_writes_lt_1ms":20} I0819 06:34:35.916031 2980 maintenance_manager.cc:391] P 4c94be5808b34fc3bb2d82e3988c20f1: we have exceeded our soft memory limit (current capacity is 192.52%). However, there are no ops currently runnable which would free memory. I0819 06:34:36.166697 2980 maintenance_manager.cc:391] P 4c94be5808b34fc3bb2d82e3988c20f1: we have exceeded our soft memory limit (current capacity is 191.41%). However, there are no ops currently runnable which would free memory. I0819 06:34:36.417527 2980 maintenance_manager.cc:391] P 4c94be5808b34fc3bb2d82e3988c20f1: we have exceeded our soft memory limit (current capacity is 192.02%). However, there are no ops currently runnable which would free memory. I0819 06:34:36.583328 1150 external_mini_cluster.cc:944] Killing /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver with pid 2847 W0819 06:34:36.654007 2798 connection.cc:551] client connection to 127.4.126.1:40980 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:36.654028 2784 connection.cc:551] client connection to 127.4.126.1:40980 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0819 06:34:36.654731 1150 external_mini_cluster.cc:944] Killing /tmp/run_tha_test479zHR/build/tsan/bin/kudu-master with pid 2785 W0819 06:34:36.664589 2783 connection.cc:551] client connection to 127.4.126.1:40980 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:36.682405 2784 connection.cc:551] client connection to 127.4.126.254:39962 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) [ OK ] RaftConsensusITest.TestEarlyCommitDespiteMemoryPressure (16639 ms) [ RUN ] RaftConsensusITest.TestEvictAbandonedFollowers I0819 06:34:36.698966 1150 test_util.cc:185] Using random seed: 1887215927 I0819 06:34:36.702183 1150 ts_itest-base.h:81] Starting cluster with: I0819 06:34:36.702283 1150 ts_itest-base.h:82] -------------- I0819 06:34:36.702358 1150 ts_itest-base.h:83] 3 tablet servers I0819 06:34:36.702430 1150 ts_itest-base.h:84] 3 replicas per TS I0819 06:34:36.702503 1150 ts_itest-base.h:85] -------------- I0819 06:34:36.704576 1150 external_mini_cluster.cc:752] Running /tmp/run_tha_test479zHR/build/tsan/bin/kudu-master /tmp/run_tha_test479zHR/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.126.254:0 --webserver_interface=127.4.126.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=flag --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --master_add_server_when_underreplicated=false with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0819 06:34:36.835019 3281 flags.cc:384] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0819 06:34:36.835398 3281 flags.cc:384] Enabled unsafe flag: --never_fsync=true W0819 06:34:36.851151 3281 flags.cc:384] Enabled experimental flag: --ipki_ca_key_size=1024 W0819 06:34:36.851300 3281 flags.cc:384] Enabled experimental flag: --ipki_server_key_size=1024 W0819 06:34:36.851394 3281 flags.cc:384] Enabled experimental flag: --tsk_num_rsa_bits=512 I0819 06:34:36.872197 3281 master_main.cc:60] Master server non-default flags: --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/master-0/wal --master_add_server_when_underreplicated=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.126.254:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.4.126.254 --webserver_port=0 --never_fsync=true --heap_profile_path=/tmp/kudu-master.3281 --redact=flag --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.5.0-SNAPSHOT revision ba0f21e2316b78f167bbaba44cab392558d97b6f build type FASTDEBUG built by None at 19 Aug 2017 06:31:02 UTC on jenkins-slave-c3oe build id 9422 TSAN enabled I0819 06:34:36.881817 3281 master_main.cc:67] Initializing master server... I0819 06:34:36.883427 3281 system_ntp.cc:96] NTP initialized. Resolution in nanos?: 1 Skew PPM: 500 Current error: 1312405 I0819 06:34:36.884335 3281 env_posix.cc:1586] Not raising process file limit of 65536; it is already as high as it can go I0819 06:34:36.884624 3281 file_cache.cc:464] Constructed file cache lbm with capacity 26214 I0819 06:34:36.889600 3281 server_base.cc:215] Could not load existing FS layout: Not found: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0819 06:34:36.889741 3281 server_base.cc:216] Creating new FS layout I0819 06:34:36.897200 3281 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data/instance: uuid: "76cb7e6ad65e4f9d8069c61523692b43" format_stamp: "Formatted at 2017-08-19 06:34:36 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:36.897955 3281 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "76cb7e6ad65e4f9d8069c61523692b43" format_stamp: "Formatted at 2017-08-19 06:34:36 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:36.908500 3281 fs_manager.cc:280] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s I0819 06:34:36.908638 3281 fs_manager.cc:283] Opened local filesystem: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data,/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/master-0/wal uuid: "76cb7e6ad65e4f9d8069c61523692b43" format_stamp: "Formatted at 2017-08-19 06:34:36 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:36.908816 3281 fs_report.cc:345] Block manager report -------------------- 1 data directories: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0819 06:34:37.089450 3281 master_main.cc:70] Starting Master server... I0819 06:34:37.090677 3298 process_memory.cc:172] Process hard memory limit is 11.777661 GB I0819 06:34:37.090834 3298 process_memory.cc:174] Process soft memory limit is 9.422129 GB I0819 06:34:37.090929 3298 process_memory.cc:177] Process memory pressure threshold is 7.066597 GB I0819 06:34:37.131956 3281 rpc_server.cc:186] RPC server started. Bound to: 127.4.126.254:34857 I0819 06:34:37.171262 3281 webserver.cc:145] Starting webserver on 127.4.126.254:0 I0819 06:34:37.171376 3281 webserver.cc:156] Document root disabled I0819 06:34:37.172684 3281 webserver.cc:278] Webserver started. Bound to: http://127.4.126.254:37040/ I0819 06:34:37.173478 3281 server_base.cc:359] Dumped server information to /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/master-0/data/info.pb I0819 06:34:37.173965 1150 external_mini_cluster.cc:814] Started /tmp/run_tha_test479zHR/build/tsan/bin/kudu-master as pid 3281 I0819 06:34:37.175022 1150 external_mini_cluster.cc:752] Running /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data --rpc_bind_addresses=127.4.126.1:0 --local_ip_for_outbound_sockets=127.4.126.1 --webserver_interface=127.4.126.1 --webserver_port=0 --tserver_master_addrs=127.4.126.254:34857 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=flag --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --max_cell_size_bytes=1000000 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 with env {} I0819 06:34:37.195981 3342 tablet_bootstrap.cc:402] T 00000000000000000000000000000000 P 76cb7e6ad65e4f9d8069c61523692b43: Bootstrap starting. I0819 06:34:37.198459 3342 tablet_bootstrap.cc:537] T 00000000000000000000000000000000 P 76cb7e6ad65e4f9d8069c61523692b43: No blocks or log segments found. Creating new log. I0819 06:34:37.199707 3342 log.cc:511] T 00000000000000000000000000000000 P 76cb7e6ad65e4f9d8069c61523692b43: Log is configured to *not* fsync() on all Append() calls I0819 06:34:37.202229 3342 tablet_bootstrap.cc:402] T 00000000000000000000000000000000 P 76cb7e6ad65e4f9d8069c61523692b43: No bootstrap required, opened a new log I0819 06:34:37.204840 3342 raft_consensus.cc:286] T 00000000000000000000000000000000 P 76cb7e6ad65e4f9d8069c61523692b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cb7e6ad65e4f9d8069c61523692b43" member_type: VOTER } I0819 06:34:37.205338 3342 raft_consensus.cc:318] T 00000000000000000000000000000000 P 76cb7e6ad65e4f9d8069c61523692b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0819 06:34:37.205494 3342 raft_consensus.cc:574] T 00000000000000000000000000000000 P 76cb7e6ad65e4f9d8069c61523692b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76cb7e6ad65e4f9d8069c61523692b43, State: Running, Role: FOLLOWER I0819 06:34:37.205708 3342 consensus_queue.cc:190] T 00000000000000000000000000000000 P 76cb7e6ad65e4f9d8069c61523692b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0819 06:34:37.205940 3342 raft_consensus.cc:329] T 00000000000000000000000000000000 P 76cb7e6ad65e4f9d8069c61523692b43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0819 06:34:37.206137 3342 raft_consensus.cc:413] T 00000000000000000000000000000000 P 76cb7e6ad65e4f9d8069c61523692b43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0819 06:34:37.206297 3342 raft_consensus.cc:2401] T 00000000000000000000000000000000 P 76cb7e6ad65e4f9d8069c61523692b43: Snoozing failure detection for election timeout plus an additional 1.601s I0819 06:34:37.206439 3342 raft_consensus.cc:2448] T 00000000000000000000000000000000 P 76cb7e6ad65e4f9d8069c61523692b43 [term 0 FOLLOWER]: Advancing to term 1 I0819 06:34:37.209110 3342 raft_consensus.cc:437] T 00000000000000000000000000000000 P 76cb7e6ad65e4f9d8069c61523692b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cb7e6ad65e4f9d8069c61523692b43" member_type: VOTER } I0819 06:34:37.209564 3342 leader_election.cc:244] T 00000000000000000000000000000000 P 76cb7e6ad65e4f9d8069c61523692b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0819 06:34:37.211321 3346 raft_consensus.cc:2401] T 00000000000000000000000000000000 P 76cb7e6ad65e4f9d8069c61523692b43: Snoozing failure detection for election timeout plus an additional 2.223s I0819 06:34:37.211624 3346 raft_consensus.cc:2205] T 00000000000000000000000000000000 P 76cb7e6ad65e4f9d8069c61523692b43 [term 1 FOLLOWER]: Leader election won for term 1 I0819 06:34:37.213893 3346 raft_consensus.cc:540] T 00000000000000000000000000000000 P 76cb7e6ad65e4f9d8069c61523692b43 [term 1 LEADER]: Becoming Leader. State: Replica: 76cb7e6ad65e4f9d8069c61523692b43, State: Running, Role: LEADER I0819 06:34:37.214179 3346 consensus_queue.cc:169] T 00000000000000000000000000000000 P 76cb7e6ad65e4f9d8069c61523692b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cb7e6ad65e4f9d8069c61523692b43" member_type: VOTER } I0819 06:34:37.229604 3342 sys_catalog.cc:287] T 00000000000000000000000000000000 P 76cb7e6ad65e4f9d8069c61523692b43 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "76cb7e6ad65e4f9d8069c61523692b43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cb7e6ad65e4f9d8069c61523692b43" member_type: VOTER } } I0819 06:34:37.229758 3348 sys_catalog.cc:287] T 00000000000000000000000000000000 P 76cb7e6ad65e4f9d8069c61523692b43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76cb7e6ad65e4f9d8069c61523692b43. Latest consensus state: current_term: 1 leader_uuid: "76cb7e6ad65e4f9d8069c61523692b43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cb7e6ad65e4f9d8069c61523692b43" member_type: VOTER } } I0819 06:34:37.230036 3342 sys_catalog.cc:290] T 00000000000000000000000000000000 P 76cb7e6ad65e4f9d8069c61523692b43 [sys.catalog]: This master's current role is: LEADER I0819 06:34:37.230130 3348 sys_catalog.cc:290] T 00000000000000000000000000000000 P 76cb7e6ad65e4f9d8069c61523692b43 [sys.catalog]: This master's current role is: LEADER I0819 06:34:37.233175 3347 sys_catalog.cc:287] T 00000000000000000000000000000000 P 76cb7e6ad65e4f9d8069c61523692b43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76cb7e6ad65e4f9d8069c61523692b43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cb7e6ad65e4f9d8069c61523692b43" member_type: VOTER } } I0819 06:34:37.234138 3347 sys_catalog.cc:290] T 00000000000000000000000000000000 P 76cb7e6ad65e4f9d8069c61523692b43 [sys.catalog]: This master's current role is: LEADER I0819 06:34:37.237179 3342 sys_catalog.cc:369] T 00000000000000000000000000000000 P 76cb7e6ad65e4f9d8069c61523692b43 [sys.catalog]: configured and running, proceeding with master startup. I0819 06:34:37.245949 3351 catalog_manager.cc:1018] Loading table and tablet metadata into memory... I0819 06:34:37.247808 3281 master_main.cc:73] Master server successfully started. I0819 06:34:37.250860 3351 catalog_manager.cc:1029] Initializing Kudu internal certificate authority... WARNING: Logging before InitGoogleLogging() is written to STDERR W0819 06:34:37.304846 3343 flags.cc:384] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0819 06:34:37.305263 3343 flags.cc:384] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0819 06:34:37.305402 3343 flags.cc:384] Enabled unsafe flag: --never_fsync=true I0819 06:34:37.307405 3351 catalog_manager.cc:914] Generated new certificate authority record I0819 06:34:37.309746 3351 catalog_manager.cc:1038] Loading token signing keys... W0819 06:34:37.319905 3343 flags.cc:384] Enabled experimental flag: --log_compression_codec=none W0819 06:34:37.320066 3343 flags.cc:384] Enabled experimental flag: --log_max_segments_to_retain=3 W0819 06:34:37.320264 3343 flags.cc:384] Enabled experimental flag: --ipki_server_key_size=1024 W0819 06:34:37.320605 3343 flags.cc:384] Enabled experimental flag: --log_target_replay_size_mb=1 W0819 06:34:37.320742 3343 flags.cc:384] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.126.1 I0819 06:34:37.321264 3351 catalog_manager.cc:3605] T 00000000000000000000000000000000 P 76cb7e6ad65e4f9d8069c61523692b43: Generated new TSK 0 I0819 06:34:37.372752 3343 tablet_server_main.cc:64] Tablet server non-default flags: --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.126.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.4.126.1 --webserver_port=0 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.4.126.254:34857 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.3343 --redact=flag --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.126.1 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.5.0-SNAPSHOT revision ba0f21e2316b78f167bbaba44cab392558d97b6f build type FASTDEBUG built by None at 19 Aug 2017 06:31:02 UTC on jenkins-slave-c3oe build id 9422 TSAN enabled I0819 06:34:37.404990 3343 tablet_server_main.cc:71] Initializing tablet server... I0819 06:34:37.406899 3343 system_ntp.cc:96] NTP initialized. Resolution in nanos?: 1 Skew PPM: 500 Current error: 1312905 I0819 06:34:37.407569 3343 env_posix.cc:1586] Not raising process file limit of 65536; it is already as high as it can go I0819 06:34:37.407866 3343 file_cache.cc:464] Constructed file cache lbm with capacity 26214 I0819 06:34:37.412962 3343 server_base.cc:215] Could not load existing FS layout: Not found: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0819 06:34:37.413105 3343 server_base.cc:216] Creating new FS layout I0819 06:34:37.420651 3343 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "cd3b50a0af2548c5b2b14337d8f9b265" format_stamp: "Formatted at 2017-08-19 06:34:37 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:37.421449 3343 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "cd3b50a0af2548c5b2b14337d8f9b265" format_stamp: "Formatted at 2017-08-19 06:34:37 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:37.453604 3343 fs_manager.cc:280] Time spent opening block manager: real 0.025s user 0.004s sys 0.024s I0819 06:34:37.453788 3343 fs_manager.cc:283] Opened local filesystem: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data,/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/wal uuid: "cd3b50a0af2548c5b2b14337d8f9b265" format_stamp: "Formatted at 2017-08-19 06:34:37 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:37.454020 3343 fs_report.cc:345] Block manager report -------------------- 1 data directories: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0819 06:34:37.686867 3343 ts_tablet_manager.cc:190] Loaded tablet metadata (0 live tablets) I0819 06:34:37.688438 3343 tablet_server_main.cc:76] Starting tablet server... I0819 06:34:37.797196 3343 rpc_server.cc:186] RPC server started. Bound to: 127.4.126.1:38321 I0819 06:34:37.798107 3343 webserver.cc:145] Starting webserver on 127.4.126.1:0 I0819 06:34:37.798218 3343 webserver.cc:156] Document root disabled I0819 06:34:37.799741 3343 webserver.cc:278] Webserver started. Bound to: http://127.4.126.1:34670/ I0819 06:34:37.800577 3343 server_base.cc:359] Dumped server information to /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-0/data/info.pb I0819 06:34:37.807503 1150 external_mini_cluster.cc:814] Started /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver as pid 3343 I0819 06:34:37.807983 3477 process_memory.cc:172] Process hard memory limit is 11.777661 GB I0819 06:34:37.808792 3477 process_memory.cc:174] Process soft memory limit is 9.422129 GB I0819 06:34:37.808976 3477 process_memory.cc:177] Process memory pressure threshold is 7.066597 GB I0819 06:34:37.808924 1150 external_mini_cluster.cc:752] Running /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data --rpc_bind_addresses=127.4.126.2:0 --local_ip_for_outbound_sockets=127.4.126.2 --webserver_interface=127.4.126.2 --webserver_port=0 --tserver_master_addrs=127.4.126.254:34857 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=flag --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --max_cell_size_bytes=1000000 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 with env {} I0819 06:34:37.808713 3343 tablet_server_main.cc:79] Tablet server successfully started. I0819 06:34:37.841009 3476 heartbeater.cc:295] Connected to a master server at 127.4.126.254:34857 I0819 06:34:37.841269 3476 heartbeater.cc:365] Registering TS with master... I0819 06:34:37.841701 3476 heartbeater.cc:392] Master 127.4.126.254:34857 requested a full tablet report, sending... I0819 06:34:37.843255 3309 ts_manager.cc:79] Registered new tserver with Master: cd3b50a0af2548c5b2b14337d8f9b265 (127.4.126.1:38321) I0819 06:34:37.844794 3309 master_service.cc:197] Signed X509 certificate for tserver {username='slave'} at 127.4.126.1:45288 WARNING: Logging before InitGoogleLogging() is written to STDERR W0819 06:34:38.010824 3480 flags.cc:384] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0819 06:34:38.011339 3480 flags.cc:384] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0819 06:34:38.011515 3480 flags.cc:384] Enabled unsafe flag: --never_fsync=true W0819 06:34:38.035070 3480 flags.cc:384] Enabled experimental flag: --log_compression_codec=none W0819 06:34:38.035285 3480 flags.cc:384] Enabled experimental flag: --log_max_segments_to_retain=3 W0819 06:34:38.035568 3480 flags.cc:384] Enabled experimental flag: --ipki_server_key_size=1024 W0819 06:34:38.036038 3480 flags.cc:384] Enabled experimental flag: --log_target_replay_size_mb=1 W0819 06:34:38.036238 3480 flags.cc:384] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.126.2 I0819 06:34:38.068208 3480 tablet_server_main.cc:64] Tablet server non-default flags: --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.126.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.4.126.2 --webserver_port=0 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.4.126.254:34857 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.3480 --redact=flag --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.126.2 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.5.0-SNAPSHOT revision ba0f21e2316b78f167bbaba44cab392558d97b6f build type FASTDEBUG built by None at 19 Aug 2017 06:31:02 UTC on jenkins-slave-c3oe build id 9422 TSAN enabled I0819 06:34:38.083384 3480 tablet_server_main.cc:71] Initializing tablet server... I0819 06:34:38.085880 3480 system_ntp.cc:96] NTP initialized. Resolution in nanos?: 1 Skew PPM: 500 Current error: 1313405 I0819 06:34:38.086711 3480 env_posix.cc:1586] Not raising process file limit of 65536; it is already as high as it can go I0819 06:34:38.087121 3480 file_cache.cc:464] Constructed file cache lbm with capacity 26214 I0819 06:34:38.095654 3480 server_base.cc:215] Could not load existing FS layout: Not found: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0819 06:34:38.095865 3480 server_base.cc:216] Creating new FS layout I0819 06:34:38.108490 3480 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "b50ee9ceeb65458495ef77d5d2dccbbc" format_stamp: "Formatted at 2017-08-19 06:34:38 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:38.109539 3480 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "b50ee9ceeb65458495ef77d5d2dccbbc" format_stamp: "Formatted at 2017-08-19 06:34:38 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:38.125630 3480 fs_manager.cc:280] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s I0819 06:34:38.125856 3480 fs_manager.cc:283] Opened local filesystem: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data,/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/wal uuid: "b50ee9ceeb65458495ef77d5d2dccbbc" format_stamp: "Formatted at 2017-08-19 06:34:38 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:38.126128 3480 fs_report.cc:345] Block manager report -------------------- 1 data directories: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0819 06:34:38.330392 3480 ts_tablet_manager.cc:190] Loaded tablet metadata (0 live tablets) I0819 06:34:38.331765 3480 tablet_server_main.cc:76] Starting tablet server... I0819 06:34:38.450558 3480 rpc_server.cc:186] RPC server started. Bound to: 127.4.126.2:51392 I0819 06:34:38.451439 3480 webserver.cc:145] Starting webserver on 127.4.126.2:0 I0819 06:34:38.451550 3480 webserver.cc:156] Document root disabled I0819 06:34:38.452915 3480 webserver.cc:278] Webserver started. Bound to: http://127.4.126.2:36824/ I0819 06:34:38.453641 3480 server_base.cc:359] Dumped server information to /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/data/info.pb I0819 06:34:38.458112 1150 external_mini_cluster.cc:814] Started /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver as pid 3480 I0819 06:34:38.459466 1150 external_mini_cluster.cc:752] Running /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data --rpc_bind_addresses=127.4.126.3:0 --local_ip_for_outbound_sockets=127.4.126.3 --webserver_interface=127.4.126.3 --webserver_port=0 --tserver_master_addrs=127.4.126.254:34857 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=flag --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --max_cell_size_bytes=1000000 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 with env {} I0819 06:34:38.462641 3602 process_memory.cc:172] Process hard memory limit is 11.777661 GB I0819 06:34:38.462823 3602 process_memory.cc:174] Process soft memory limit is 9.422129 GB I0819 06:34:38.462923 3602 process_memory.cc:177] Process memory pressure threshold is 7.066597 GB I0819 06:34:38.464025 3480 tablet_server_main.cc:79] Tablet server successfully started. I0819 06:34:38.494125 3601 heartbeater.cc:295] Connected to a master server at 127.4.126.254:34857 I0819 06:34:38.494478 3601 heartbeater.cc:365] Registering TS with master... I0819 06:34:38.495061 3601 heartbeater.cc:392] Master 127.4.126.254:34857 requested a full tablet report, sending... I0819 06:34:38.496848 3309 ts_manager.cc:79] Registered new tserver with Master: b50ee9ceeb65458495ef77d5d2dccbbc (127.4.126.2:51392) I0819 06:34:38.498433 3309 master_service.cc:197] Signed X509 certificate for tserver {username='slave'} at 127.4.126.2:45285 WARNING: Logging before InitGoogleLogging() is written to STDERR W0819 06:34:38.596071 3605 flags.cc:384] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0819 06:34:38.596493 3605 flags.cc:384] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0819 06:34:38.596624 3605 flags.cc:384] Enabled unsafe flag: --never_fsync=true W0819 06:34:38.611130 3605 flags.cc:384] Enabled experimental flag: --log_compression_codec=none W0819 06:34:38.611306 3605 flags.cc:384] Enabled experimental flag: --log_max_segments_to_retain=3 W0819 06:34:38.611497 3605 flags.cc:384] Enabled experimental flag: --ipki_server_key_size=1024 W0819 06:34:38.611799 3605 flags.cc:384] Enabled experimental flag: --log_target_replay_size_mb=1 W0819 06:34:38.611929 3605 flags.cc:384] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.126.3 I0819 06:34:38.631773 3605 tablet_server_main.cc:64] Tablet server non-default flags: --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.126.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.4.126.3 --webserver_port=0 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.4.126.254:34857 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.3605 --redact=flag --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.126.3 --log_dir=/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.5.0-SNAPSHOT revision ba0f21e2316b78f167bbaba44cab392558d97b6f build type FASTDEBUG built by None at 19 Aug 2017 06:31:02 UTC on jenkins-slave-c3oe build id 9422 TSAN enabled I0819 06:34:38.641809 3605 tablet_server_main.cc:71] Initializing tablet server... I0819 06:34:38.643666 3605 system_ntp.cc:96] NTP initialized. Resolution in nanos?: 1 Skew PPM: 500 Current error: 1313405 I0819 06:34:38.644311 3605 env_posix.cc:1586] Not raising process file limit of 65536; it is already as high as it can go I0819 06:34:38.644603 3605 file_cache.cc:464] Constructed file cache lbm with capacity 26214 I0819 06:34:38.649715 3605 server_base.cc:215] Could not load existing FS layout: Not found: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0819 06:34:38.649857 3605 server_base.cc:216] Creating new FS layout I0819 06:34:38.657289 3605 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "a331dc44a9014f3caa72b13d0954dd6c" format_stamp: "Formatted at 2017-08-19 06:34:38 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:38.658012 3605 fs_manager.cc:394] Generated new instance metadata in path /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "a331dc44a9014f3caa72b13d0954dd6c" format_stamp: "Formatted at 2017-08-19 06:34:38 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:38.668597 3605 fs_manager.cc:280] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s I0819 06:34:38.668740 3605 fs_manager.cc:283] Opened local filesystem: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data,/tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/wal uuid: "a331dc44a9014f3caa72b13d0954dd6c" format_stamp: "Formatted at 2017-08-19 06:34:38 on dist-test-slave-dist-test-slave-j33z" I0819 06:34:38.668922 3605 fs_report.cc:345] Block manager report -------------------- 1 data directories: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0819 06:34:38.847188 3476 heartbeater.cc:384] Master 127.4.126.254:34857 was elected leader, sending a full tablet report... I0819 06:34:38.883749 3605 ts_tablet_manager.cc:190] Loaded tablet metadata (0 live tablets) I0819 06:34:38.884825 3605 tablet_server_main.cc:76] Starting tablet server... I0819 06:34:38.976019 3605 rpc_server.cc:186] RPC server started. Bound to: 127.4.126.3:50890 I0819 06:34:38.976927 3605 webserver.cc:145] Starting webserver on 127.4.126.3:0 I0819 06:34:38.977030 3605 webserver.cc:156] Document root disabled I0819 06:34:38.978389 3605 webserver.cc:278] Webserver started. Bound to: http://127.4.126.3:55741/ I0819 06:34:38.979104 3605 server_base.cc:359] Dumped server information to /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/data/info.pb I0819 06:34:38.982609 3727 process_memory.cc:172] Process hard memory limit is 11.777661 GB I0819 06:34:38.983042 1150 external_mini_cluster.cc:814] Started /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver as pid 3605 I0819 06:34:38.983196 3727 process_memory.cc:174] Process soft memory limit is 9.422129 GB I0819 06:34:38.983346 3727 process_memory.cc:177] Process memory pressure threshold is 7.066597 GB I0819 06:34:38.983120 3605 tablet_server_main.cc:79] Tablet server successfully started. I0819 06:34:39.032258 3726 heartbeater.cc:295] Connected to a master server at 127.4.126.254:34857 I0819 06:34:39.032527 3726 heartbeater.cc:365] Registering TS with master... I0819 06:34:39.033038 3726 heartbeater.cc:392] Master 127.4.126.254:34857 requested a full tablet report, sending... I0819 06:34:39.034474 3309 ts_manager.cc:79] Registered new tserver with Master: a331dc44a9014f3caa72b13d0954dd6c (127.4.126.3:50890) I0819 06:34:39.035689 3309 master_service.cc:197] Signed X509 certificate for tserver {username='slave'} at 127.4.126.3:40807 I0819 06:34:39.035809 1150 external_mini_cluster.cc:454] 3 TS(s) registered with all masters I0819 06:34:39.054087 3309 catalog_manager.cc:1281] CreateTable from {username='slave'} at 127.0.0.1:41593: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0819 06:34:39.096565 3538 tablet_service.cc:684] Processing CreateTablet for tablet baa240f65f74450e95664c3b5adb6da9 (table=TestTable [id=ea5aa20913aa4e8197eb17aef1cb6962]), partition=RANGE (key) PARTITION UNBOUNDED I0819 06:34:39.103400 3663 tablet_service.cc:684] Processing CreateTablet for tablet baa240f65f74450e95664c3b5adb6da9 (table=TestTable [id=ea5aa20913aa4e8197eb17aef1cb6962]), partition=RANGE (key) PARTITION UNBOUNDED I0819 06:34:39.108988 3538 ts_tablet_manager.cc:877] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Registered tablet (data state: TABLET_DATA_READY) I0819 06:34:39.117431 3413 tablet_service.cc:684] Processing CreateTablet for tablet baa240f65f74450e95664c3b5adb6da9 (table=TestTable [id=ea5aa20913aa4e8197eb17aef1cb6962]), partition=RANGE (key) PARTITION UNBOUNDED I0819 06:34:39.123419 3663 ts_tablet_manager.cc:877] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Registered tablet (data state: TABLET_DATA_READY) I0819 06:34:39.123556 3746 ts_tablet_manager.cc:740] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Bootstrapping tablet I0819 06:34:39.124195 3746 tablet_bootstrap.cc:402] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Bootstrap starting. I0819 06:34:39.127254 3746 tablet_bootstrap.cc:537] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: No blocks or log segments found. Creating new log. I0819 06:34:39.129891 3413 ts_tablet_manager.cc:877] T baa240f65f74450e95664c3b5adb6da9 P cd3b50a0af2548c5b2b14337d8f9b265: Registered tablet (data state: TABLET_DATA_READY) I0819 06:34:39.133244 3747 ts_tablet_manager.cc:740] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Bootstrapping tablet I0819 06:34:39.133747 3747 tablet_bootstrap.cc:402] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Bootstrap starting. I0819 06:34:39.137040 3747 tablet_bootstrap.cc:537] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: No blocks or log segments found. Creating new log. I0819 06:34:39.138098 3746 log.cc:511] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Log is configured to *not* fsync() on all Append() calls I0819 06:34:39.138646 3747 log.cc:511] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Log is configured to *not* fsync() on all Append() calls I0819 06:34:39.141705 3748 ts_tablet_manager.cc:740] T baa240f65f74450e95664c3b5adb6da9 P cd3b50a0af2548c5b2b14337d8f9b265: Bootstrapping tablet I0819 06:34:39.141841 3747 tablet_bootstrap.cc:402] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: No bootstrap required, opened a new log I0819 06:34:39.142159 3747 ts_tablet_manager.cc:752] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s I0819 06:34:39.142179 3748 tablet_bootstrap.cc:402] T baa240f65f74450e95664c3b5adb6da9 P cd3b50a0af2548c5b2b14337d8f9b265: Bootstrap starting. I0819 06:34:39.144708 3748 tablet_bootstrap.cc:537] T baa240f65f74450e95664c3b5adb6da9 P cd3b50a0af2548c5b2b14337d8f9b265: No blocks or log segments found. Creating new log. I0819 06:34:39.146004 3748 log.cc:511] T baa240f65f74450e95664c3b5adb6da9 P cd3b50a0af2548c5b2b14337d8f9b265: Log is configured to *not* fsync() on all Append() calls I0819 06:34:39.149502 3746 tablet_bootstrap.cc:402] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: No bootstrap required, opened a new log I0819 06:34:39.149931 3746 ts_tablet_manager.cc:752] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s I0819 06:34:39.153475 3747 raft_consensus.cc:286] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3b50a0af2548c5b2b14337d8f9b265" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 38321 } } peers { permanent_uuid: "b50ee9ceeb65458495ef77d5d2dccbbc" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 51392 } } peers { permanent_uuid: "a331dc44a9014f3caa72b13d0954dd6c" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 50890 } } I0819 06:34:39.154613 3747 raft_consensus.cc:318] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0819 06:34:39.152899 3746 raft_consensus.cc:286] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3b50a0af2548c5b2b14337d8f9b265" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 38321 } } peers { permanent_uuid: "b50ee9ceeb65458495ef77d5d2dccbbc" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 51392 } } peers { permanent_uuid: "a331dc44a9014f3caa72b13d0954dd6c" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 50890 } } I0819 06:34:39.154899 3747 raft_consensus.cc:574] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a331dc44a9014f3caa72b13d0954dd6c, State: Running, Role: FOLLOWER I0819 06:34:39.155037 3746 raft_consensus.cc:318] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0819 06:34:39.155231 3746 raft_consensus.cc:574] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b50ee9ceeb65458495ef77d5d2dccbbc, State: Running, Role: FOLLOWER I0819 06:34:39.155231 3747 consensus_queue.cc:190] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0819 06:34:39.155504 3746 consensus_queue.cc:190] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0819 06:34:39.162273 3726 heartbeater.cc:384] Master 127.4.126.254:34857 was elected leader, sending a full tablet report... I0819 06:34:39.162395 3601 heartbeater.cc:384] Master 127.4.126.254:34857 was elected leader, sending a full tablet report... I0819 06:34:39.163591 3748 tablet_bootstrap.cc:402] T baa240f65f74450e95664c3b5adb6da9 P cd3b50a0af2548c5b2b14337d8f9b265: No bootstrap required, opened a new log I0819 06:34:39.164999 3748 ts_tablet_manager.cc:752] T baa240f65f74450e95664c3b5adb6da9 P cd3b50a0af2548c5b2b14337d8f9b265: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.012s I0819 06:34:39.172432 3747 ts_tablet_manager.cc:781] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Time spent starting tablet: real 0.030s user 0.000s sys 0.020s I0819 06:34:39.189380 3748 raft_consensus.cc:286] T baa240f65f74450e95664c3b5adb6da9 P cd3b50a0af2548c5b2b14337d8f9b265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3b50a0af2548c5b2b14337d8f9b265" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 38321 } } peers { permanent_uuid: "b50ee9ceeb65458495ef77d5d2dccbbc" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 51392 } } peers { permanent_uuid: "a331dc44a9014f3caa72b13d0954dd6c" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 50890 } } I0819 06:34:39.190207 3748 raft_consensus.cc:318] T baa240f65f74450e95664c3b5adb6da9 P cd3b50a0af2548c5b2b14337d8f9b265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0819 06:34:39.190374 3748 raft_consensus.cc:574] T baa240f65f74450e95664c3b5adb6da9 P cd3b50a0af2548c5b2b14337d8f9b265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3b50a0af2548c5b2b14337d8f9b265, State: Running, Role: FOLLOWER I0819 06:34:39.190603 3748 consensus_queue.cc:190] T baa240f65f74450e95664c3b5adb6da9 P cd3b50a0af2548c5b2b14337d8f9b265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0819 06:34:39.198694 3748 ts_tablet_manager.cc:781] T baa240f65f74450e95664c3b5adb6da9 P cd3b50a0af2548c5b2b14337d8f9b265: Time spent starting tablet: real 0.033s user 0.000s sys 0.032s I0819 06:34:39.162834 3746 ts_tablet_manager.cc:781] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Time spent starting tablet: real 0.013s user 0.004s sys 0.008s I0819 06:34:39.324609 3753 raft_consensus.cc:413] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0819 06:34:39.324875 3753 raft_consensus.cc:2401] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Snoozing failure detection for election timeout plus an additional 1.727s I0819 06:34:39.325059 3753 raft_consensus.cc:437] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3b50a0af2548c5b2b14337d8f9b265" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 38321 } } peers { permanent_uuid: "b50ee9ceeb65458495ef77d5d2dccbbc" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 51392 } } peers { permanent_uuid: "a331dc44a9014f3caa72b13d0954dd6c" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 50890 } } I0819 06:34:39.326189 3753 leader_election.cc:217] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer cd3b50a0af2548c5b2b14337d8f9b265 I0819 06:34:39.326670 3753 leader_election.cc:217] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a331dc44a9014f3caa72b13d0954dd6c I0819 06:34:39.351791 3433 raft_consensus.cc:2401] T baa240f65f74450e95664c3b5adb6da9 P cd3b50a0af2548c5b2b14337d8f9b265: Snoozing failure detection for election timeout plus an additional 1.558s I0819 06:34:39.352037 3433 raft_consensus.cc:1967] T baa240f65f74450e95664c3b5adb6da9 P cd3b50a0af2548c5b2b14337d8f9b265 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b50ee9ceeb65458495ef77d5d2dccbbc in term 0. I0819 06:34:39.353065 3490 leader_election.cc:349] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc [CANDIDATE]: Term 1 pre-election: Vote granted by peer cd3b50a0af2548c5b2b14337d8f9b265 I0819 06:34:39.353301 3490 leader_election.cc:244] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0819 06:34:39.353786 3755 raft_consensus.cc:2401] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Snoozing failure detection for election timeout plus an additional 2.124s I0819 06:34:39.354007 3755 raft_consensus.cc:2205] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc [term 0 FOLLOWER]: Leader pre-election won for term 1 I0819 06:34:39.354149 3755 raft_consensus.cc:413] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0819 06:34:39.354326 3755 raft_consensus.cc:2401] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Snoozing failure detection for election timeout plus an additional 2.000s I0819 06:34:39.354293 3683 raft_consensus.cc:2401] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Snoozing failure detection for election timeout plus an additional 1.883s I0819 06:34:39.354480 3755 raft_consensus.cc:2448] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc [term 0 FOLLOWER]: Advancing to term 1 I0819 06:34:39.354544 3683 raft_consensus.cc:1967] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b50ee9ceeb65458495ef77d5d2dccbbc in term 0. I0819 06:34:39.355677 3491 leader_election.cc:349] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc [CANDIDATE]: Term 1 pre-election: Vote granted by peer a331dc44a9014f3caa72b13d0954dd6c I0819 06:34:39.357777 3755 raft_consensus.cc:437] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3b50a0af2548c5b2b14337d8f9b265" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 38321 } } peers { permanent_uuid: "b50ee9ceeb65458495ef77d5d2dccbbc" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 51392 } } peers { permanent_uuid: "a331dc44a9014f3caa72b13d0954dd6c" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 50890 } } I0819 06:34:39.358639 3755 leader_election.cc:217] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc [CANDIDATE]: Term 1 election: Requesting vote from peer cd3b50a0af2548c5b2b14337d8f9b265 I0819 06:34:39.358968 3755 leader_election.cc:217] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc [CANDIDATE]: Term 1 election: Requesting vote from peer a331dc44a9014f3caa72b13d0954dd6c I0819 06:34:39.359992 3433 raft_consensus.cc:2448] T baa240f65f74450e95664c3b5adb6da9 P cd3b50a0af2548c5b2b14337d8f9b265 [term 0 FOLLOWER]: Advancing to term 1 I0819 06:34:39.360285 3433 raft_consensus.cc:2401] T baa240f65f74450e95664c3b5adb6da9 P cd3b50a0af2548c5b2b14337d8f9b265: Snoozing failure detection for election timeout plus an additional 2.174s I0819 06:34:39.360329 3683 raft_consensus.cc:2448] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c [term 0 FOLLOWER]: Advancing to term 1 I0819 06:34:39.360738 3683 raft_consensus.cc:2401] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Snoozing failure detection for election timeout plus an additional 1.527s I0819 06:34:39.363011 3433 raft_consensus.cc:1967] T baa240f65f74450e95664c3b5adb6da9 P cd3b50a0af2548c5b2b14337d8f9b265 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b50ee9ceeb65458495ef77d5d2dccbbc in term 1. I0819 06:34:39.363759 3490 leader_election.cc:349] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc [CANDIDATE]: Term 1 election: Vote granted by peer cd3b50a0af2548c5b2b14337d8f9b265 I0819 06:34:39.363919 3490 leader_election.cc:244] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0819 06:34:39.364225 3755 raft_consensus.cc:2401] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Snoozing failure detection for election timeout plus an additional 1.665s I0819 06:34:39.364411 3755 raft_consensus.cc:2205] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc [term 1 FOLLOWER]: Leader election won for term 1 I0819 06:34:39.364765 3683 raft_consensus.cc:1967] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b50ee9ceeb65458495ef77d5d2dccbbc in term 1. I0819 06:34:39.365654 3491 leader_election.cc:349] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc [CANDIDATE]: Term 1 election: Vote granted by peer a331dc44a9014f3caa72b13d0954dd6c I0819 06:34:39.366520 3755 raft_consensus.cc:540] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc [term 1 LEADER]: Becoming Leader. State: Replica: b50ee9ceeb65458495ef77d5d2dccbbc, State: Running, Role: LEADER I0819 06:34:39.366993 3755 consensus_queue.cc:169] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3b50a0af2548c5b2b14337d8f9b265" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 38321 } } peers { permanent_uuid: "b50ee9ceeb65458495ef77d5d2dccbbc" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 51392 } } peers { permanent_uuid: "a331dc44a9014f3caa72b13d0954dd6c" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 50890 } } I0819 06:34:39.380292 3309 catalog_manager.cc:2559] T baa240f65f74450e95664c3b5adb6da9 reported consensus state change: term changed from 0 to 1, leader changed from to b50ee9ceeb65458495ef77d5d2dccbbc (127.4.126.2). New consensus state: current_term: 1 leader_uuid: "b50ee9ceeb65458495ef77d5d2dccbbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3b50a0af2548c5b2b14337d8f9b265" member_type: VOTER last_known_addr { host: "127.4.126.1" port: 38321 } } peers { permanent_uuid: "b50ee9ceeb65458495ef77d5d2dccbbc" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 51392 } } peers { permanent_uuid: "a331dc44a9014f3caa72b13d0954dd6c" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 50890 } } } I0819 06:34:39.392124 1150 external_mini_cluster.cc:454] 3 TS(s) registered with all masters I0819 06:34:39.394351 1150 ts_itest-base.h:198] Waiting for 1 tablets on tserver cd3b50a0af2548c5b2b14337d8f9b265 to finish bootstrapping I0819 06:34:39.403554 1150 ts_itest-base.h:198] Waiting for 1 tablets on tserver b50ee9ceeb65458495ef77d5d2dccbbc to finish bootstrapping I0819 06:34:39.412793 1150 ts_itest-base.h:198] Waiting for 1 tablets on tserver a331dc44a9014f3caa72b13d0954dd6c to finish bootstrapping I0819 06:34:39.447540 1150 cluster_itest_util.cc:220] Not converged past 1 yet: 0.0 1.1 0.0 I0819 06:34:39.552603 1150 cluster_itest_util.cc:220] Not converged past 1 yet: 0.0 1.1 0.0 I0819 06:34:39.754643 3683 raft_consensus.cc:961] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c [term 1 FOLLOWER]: Refusing update from remote peer b50ee9ceeb65458495ef77d5d2dccbbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0819 06:34:39.755961 3755 consensus_queue.cc:726] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc [LEADER]: Connected to new peer: Peer: a331dc44a9014f3caa72b13d0954dd6c, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I0819 06:34:39.770557 1150 cluster_itest_util.cc:220] Not converged past 1 yet: 0.0 1.1 1.1 I0819 06:34:39.775156 3433 raft_consensus.cc:961] T baa240f65f74450e95664c3b5adb6da9 P cd3b50a0af2548c5b2b14337d8f9b265 [term 1 FOLLOWER]: Refusing update from remote peer b50ee9ceeb65458495ef77d5d2dccbbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0819 06:34:39.776226 3765 consensus_queue.cc:726] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc [LEADER]: Connected to new peer: Peer: cd3b50a0af2548c5b2b14337d8f9b265, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I0819 06:34:40.079699 1150 test_util.cc:185] Using random seed: 1890596663 I0819 06:34:40.098412 1150 test_workload.cc:273] TestWorkload: Skipping table creation because table TestTable already exists I0819 06:34:40.102862 1150 raft_consensus-itest.cc:735] Waiting until we've written at least 4MB... I0819 06:34:40.237881 3761 log.cc:647] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Max segment size reached. Starting new segment allocation I0819 06:34:40.248245 3761 log.cc:562] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Rolled over to a new log segment at /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/wal/wals/baa240f65f74450e95664c3b5adb6da9/wal-000000002 I0819 06:34:40.249209 3764 log.cc:647] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Max segment size reached. Starting new segment allocation I0819 06:34:40.258594 3764 log.cc:562] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Rolled over to a new log segment at /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/wal/wals/baa240f65f74450e95664c3b5adb6da9/wal-000000002 I0819 06:34:40.317908 3761 log.cc:647] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Max segment size reached. Starting new segment allocation I0819 06:34:40.319141 3761 log.cc:562] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Rolled over to a new log segment at /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/wal/wals/baa240f65f74450e95664c3b5adb6da9/wal-000000003 I0819 06:34:40.345590 3764 log.cc:647] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Max segment size reached. Starting new segment allocation I0819 06:34:40.347121 3764 log.cc:562] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Rolled over to a new log segment at /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/wal/wals/baa240f65f74450e95664c3b5adb6da9/wal-000000003 I0819 06:34:40.369781 3486 tablet.cc:1040] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:40.369983 3486 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:40.370136 3486 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:40.370252 3486 tablet.cc:1042] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Memstore in-memory size: 4063497 bytes I0819 06:34:40.370415 3486 tablet.cc:1305] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6156797871392997376 or (T in {6156797871392997376})}] I0819 06:34:40.390326 3611 tablet.cc:1040] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:40.390522 3611 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:40.390656 3611 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:40.390770 3611 tablet.cc:1042] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Memstore in-memory size: 4063497 bytes I0819 06:34:40.390936 3611 tablet.cc:1305] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6156797871392997376 or (T in {6156797871392997376})}] I0819 06:34:40.416716 3611 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:40.429525 3486 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:40.534543 3761 log.cc:647] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Max segment size reached. Starting new segment allocation I0819 06:34:40.536202 3761 log.cc:562] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Rolled over to a new log segment at /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/wal/wals/baa240f65f74450e95664c3b5adb6da9/wal-000000004 I0819 06:34:40.544945 3764 log.cc:647] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Max segment size reached. Starting new segment allocation I0819 06:34:40.546469 3764 log.cc:562] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Rolled over to a new log segment at /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/wal/wals/baa240f65f74450e95664c3b5adb6da9/wal-000000004 I0819 06:34:40.619683 3611 tablet.cc:1390] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:40.620184 3611 tablet.cc:1445] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:40.620369 3611 tablet.cc:1447] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6156797872424992768 or (T in {6156797872424992768})}] I0819 06:34:40.629986 3611 tablet.cc:1489] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Flush successful on 16 rows (135803 bytes) I0819 06:34:40.673895 3611 maintenance_manager.cc:466] Time spent running FlushMRSOp(baa240f65f74450e95664c3b5adb6da9): real 0.284s user 0.160s sys 0.020s I0819 06:34:40.674146 3611 maintenance_manager.cc:472] P a331dc44a9014f3caa72b13d0954dd6c: FlushMRSOp(baa240f65f74450e95664c3b5adb6da9) metrics: {"cfile_init":1,"compiler_manager_pool.queue_time_us":10254,"data dir 0.queue_time_us":19952,"data dir 0.run_cpu_time_us":393,"data dir 0.run_wall_time_us":1105,"lbm_read_time_us":186,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1723,"lbm_writes_lt_1ms":23,"thread_start_us":27179,"threads_started":2} I0819 06:34:40.676127 3611 log_reader.cc:381] T baa240f65f74450e95664c3b5adb6da9: removed 1 log segments from log reader I0819 06:34:40.676442 3611 log.cc:868] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Deleting log segment in path: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/wal/wals/baa240f65f74450e95664c3b5adb6da9/wal-000000001 (ops 1-8) I0819 06:34:40.677145 3611 maintenance_manager.cc:466] Time spent running LogGCOp(baa240f65f74450e95664c3b5adb6da9): real 0.001s user 0.000s sys 0.000s I0819 06:34:40.677340 3611 maintenance_manager.cc:472] P a331dc44a9014f3caa72b13d0954dd6c: LogGCOp(baa240f65f74450e95664c3b5adb6da9) metrics: {} I0819 06:34:40.680534 3611 maintenance_manager.cc:466] Time spent running UndoDeltaBlockGCOp(baa240f65f74450e95664c3b5adb6da9): real 0.001s user 0.000s sys 0.000s I0819 06:34:40.682462 3611 maintenance_manager.cc:472] P a331dc44a9014f3caa72b13d0954dd6c: UndoDeltaBlockGCOp(baa240f65f74450e95664c3b5adb6da9) metrics: {"cfile_init":1,"lbm_read_time_us":192,"lbm_reads_lt_1ms":4} I0819 06:34:40.696159 3761 log.cc:647] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Max segment size reached. Starting new segment allocation I0819 06:34:40.697562 3761 log.cc:562] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Rolled over to a new log segment at /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/wal/wals/baa240f65f74450e95664c3b5adb6da9/wal-000000005 I0819 06:34:40.707186 3486 tablet.cc:1390] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:40.707751 3486 tablet.cc:1445] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:40.707929 3486 tablet.cc:1447] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6156797872810946560 or (T in {6156797872810946560})}] I0819 06:34:40.709640 3764 log.cc:647] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Max segment size reached. Starting new segment allocation I0819 06:34:40.711293 3764 log.cc:562] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Rolled over to a new log segment at /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/wal/wals/baa240f65f74450e95664c3b5adb6da9/wal-000000005 I0819 06:34:40.715203 3486 tablet.cc:1489] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Flush successful on 16 rows (135803 bytes) I0819 06:34:40.715857 3486 maintenance_manager.cc:466] Time spent running FlushMRSOp(baa240f65f74450e95664c3b5adb6da9): real 0.347s user 0.112s sys 0.036s I0819 06:34:40.716063 3486 maintenance_manager.cc:472] P b50ee9ceeb65458495ef77d5d2dccbbc: FlushMRSOp(baa240f65f74450e95664c3b5adb6da9) metrics: {"cfile_init":1,"compiler_manager_pool.queue_time_us":16989,"data dir 0.queue_time_us":18398,"data dir 0.run_cpu_time_us":397,"data dir 0.run_wall_time_us":1102,"lbm_read_time_us":229,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1545,"lbm_writes_lt_1ms":23,"thread_start_us":35062,"threads_started":2} I0819 06:34:40.756500 3486 tablet.cc:1040] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:40.756685 3486 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:40.756824 3486 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:40.756988 3486 tablet.cc:1042] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Memstore in-memory size: 4063497 bytes I0819 06:34:40.757464 3486 tablet.cc:1305] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6156797872921325568 or (T in {6156797872921325568})}] I0819 06:34:40.759397 3486 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:40.785769 3611 log_reader.cc:381] T baa240f65f74450e95664c3b5adb6da9: removed 1 log segments from log reader I0819 06:34:40.785965 3611 log.cc:868] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Deleting log segment in path: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/wal/wals/baa240f65f74450e95664c3b5adb6da9/wal-000000002 (ops 9-15) I0819 06:34:40.786497 3611 maintenance_manager.cc:466] Time spent running LogGCOp(baa240f65f74450e95664c3b5adb6da9): real 0.001s user 0.000s sys 0.000s I0819 06:34:40.786664 3611 maintenance_manager.cc:472] P a331dc44a9014f3caa72b13d0954dd6c: LogGCOp(baa240f65f74450e95664c3b5adb6da9) metrics: {} I0819 06:34:40.788003 3611 tablet.cc:1040] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Flush: entering stage 1 (old memrowset already frozen for inserts) I0819 06:34:40.788157 3611 compaction.cc:884] Selected 1 rowsets to compact: I0819 06:34:40.788271 3611 compaction.cc:887] memrowset(current size on disk: ~0 bytes) I0819 06:34:40.788383 3611 tablet.cc:1042] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Memstore in-memory size: 4063497 bytes I0819 06:34:40.788633 3611 tablet.cc:1305] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6156797872963424256 or (T in {6156797872963424256})}] I0819 06:34:40.790132 3611 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:40.835623 3761 log.cc:647] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Max segment size reached. Starting new segment allocation I0819 06:34:40.837106 3761 log.cc:562] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Rolled over to a new log segment at /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/wal/wals/baa240f65f74450e95664c3b5adb6da9/wal-000000006 I0819 06:34:40.868734 3764 log.cc:647] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Max segment size reached. Starting new segment allocation I0819 06:34:40.870239 3764 log.cc:562] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Rolled over to a new log segment at /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/wal/wals/baa240f65f74450e95664c3b5adb6da9/wal-000000006 I0819 06:34:40.922935 1150 raft_consensus-itest.cc:741] Waiting for log GC on b50ee9ceeb65458495ef77d5d2dccbbc I0819 06:34:40.931762 3486 tablet.cc:1390] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:40.932288 3486 tablet.cc:1445] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:40.933949 3486 tablet.cc:1447] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6156797873492078592 or (T in {6156797873492078592})}] I0819 06:34:40.947547 3486 tablet.cc:1489] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Flush successful on 13 rows (135803 bytes) I0819 06:34:40.957151 3611 tablet.cc:1390] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:40.957506 3611 tablet.cc:1445] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:40.957612 3611 tablet.cc:1447] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6156797873492078592 or (T in {6156797873492078592})}] I0819 06:34:40.963577 3611 tablet.cc:1489] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Flush successful on 14 rows (135803 bytes) I0819 06:34:40.994660 3611 maintenance_manager.cc:466] Time spent running FlushMRSOp(baa240f65f74450e95664c3b5adb6da9): real 0.207s user 0.124s sys 0.000s I0819 06:34:40.994926 3611 maintenance_manager.cc:472] P a331dc44a9014f3caa72b13d0954dd6c: FlushMRSOp(baa240f65f74450e95664c3b5adb6da9) metrics: {"cfile_init":1,"data dir 0.queue_time_us":80,"data dir 0.run_cpu_time_us":199,"data dir 0.run_wall_time_us":1352,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4,"lbm_write_time_us":13305,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22} I0819 06:34:40.996819 3611 log_reader.cc:381] T baa240f65f74450e95664c3b5adb6da9: removed 1 log segments from log reader I0819 06:34:40.997035 3611 log.cc:868] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Deleting log segment in path: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/wal/wals/baa240f65f74450e95664c3b5adb6da9/wal-000000003 (ops 16-22) I0819 06:34:40.997613 3611 maintenance_manager.cc:466] Time spent running LogGCOp(baa240f65f74450e95664c3b5adb6da9): real 0.001s user 0.000s sys 0.000s I0819 06:34:40.997778 3611 maintenance_manager.cc:472] P a331dc44a9014f3caa72b13d0954dd6c: LogGCOp(baa240f65f74450e95664c3b5adb6da9) metrics: {} I0819 06:34:40.999236 3611 maintenance_manager.cc:466] Time spent running UndoDeltaBlockGCOp(baa240f65f74450e95664c3b5adb6da9): real 0.001s user 0.000s sys 0.000s I0819 06:34:40.999410 3611 maintenance_manager.cc:472] P a331dc44a9014f3caa72b13d0954dd6c: UndoDeltaBlockGCOp(baa240f65f74450e95664c3b5adb6da9) metrics: {"cfile_init":1,"lbm_read_time_us":191,"lbm_reads_lt_1ms":4} I0819 06:34:41.000655 3611 tablet.cc:1519] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Compaction: stage 1 complete, picked 2 rowsets to compact I0819 06:34:41.000792 3486 maintenance_manager.cc:466] Time spent running FlushMRSOp(baa240f65f74450e95664c3b5adb6da9): real 0.256s user 0.116s sys 0.008s I0819 06:34:41.000849 3611 compaction.cc:884] Selected 2 rowsets to compact: I0819 06:34:41.000982 3611 compaction.cc:887] RowSet(0)(current size on disk: ~136119 bytes) I0819 06:34:41.001137 3611 compaction.cc:887] RowSet(1)(current size on disk: ~136102 bytes) I0819 06:34:41.000926 3486 maintenance_manager.cc:472] P b50ee9ceeb65458495ef77d5d2dccbbc: FlushMRSOp(baa240f65f74450e95664c3b5adb6da9) metrics: {"cfile_init":1,"data dir 0.queue_time_us":61,"data dir 0.run_cpu_time_us":240,"data dir 0.run_wall_time_us":1405,"lbm_read_time_us":162,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1763,"lbm_writes_lt_1ms":23,"spinlock_wait_cycles":512896} I0819 06:34:41.001291 3611 tablet.cc:1305] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6156797873492078592 or (T in {6156797873492078592})}] I0819 06:34:41.002745 3486 log_reader.cc:381] T baa240f65f74450e95664c3b5adb6da9: removed 3 log segments from log reader I0819 06:34:41.003023 3486 log.cc:868] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Deleting log segment in path: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/wal/wals/baa240f65f74450e95664c3b5adb6da9/wal-000000001 (ops 1-8) I0819 06:34:41.003319 3486 log.cc:868] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Deleting log segment in path: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/wal/wals/baa240f65f74450e95664c3b5adb6da9/wal-000000002 (ops 9-15) I0819 06:34:41.003324 3611 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) I0819 06:34:41.003535 3486 log.cc:868] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Deleting log segment in path: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/wal/wals/baa240f65f74450e95664c3b5adb6da9/wal-000000003 (ops 16-22) I0819 06:34:41.004552 3486 maintenance_manager.cc:466] Time spent running LogGCOp(baa240f65f74450e95664c3b5adb6da9): real 0.002s user 0.000s sys 0.000s I0819 06:34:41.004727 3486 maintenance_manager.cc:472] P b50ee9ceeb65458495ef77d5d2dccbbc: LogGCOp(baa240f65f74450e95664c3b5adb6da9) metrics: {} I0819 06:34:41.005983 1150 raft_consensus-itest.cc:746] Log GC complete on b50ee9ceeb65458495ef77d5d2dccbbc I0819 06:34:41.007292 3486 maintenance_manager.cc:466] Time spent running UndoDeltaBlockGCOp(baa240f65f74450e95664c3b5adb6da9): real 0.002s user 0.000s sys 0.000s I0819 06:34:41.007473 3486 maintenance_manager.cc:472] P b50ee9ceeb65458495ef77d5d2dccbbc: UndoDeltaBlockGCOp(baa240f65f74450e95664c3b5adb6da9) metrics: {"cfile_init":2,"lbm_read_time_us":356,"lbm_reads_lt_1ms":8} I0819 06:34:41.008780 3486 tablet.cc:1519] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Compaction: stage 1 complete, picked 2 rowsets to compact I0819 06:34:41.008977 3486 compaction.cc:884] Selected 2 rowsets to compact: I0819 06:34:41.009120 3486 compaction.cc:887] RowSet(0)(current size on disk: ~136119 bytes) I0819 06:34:41.009305 3486 compaction.cc:887] RowSet(1)(current size on disk: ~136093 bytes) I0819 06:34:41.009483 3486 tablet.cc:1305] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6156797873492078592 or (T in {6156797873492078592})}] I0819 06:34:41.011443 3486 multi_column_writer.cc:95] Opened CFile writers for 3 column(s) W0819 06:34:41.156327 3490 consensus_peers.cc:396] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc -> Peer cd3b50a0af2548c5b2b14337d8f9b265 (127.4.126.1:38321): Couldn't send request to peer cd3b50a0af2548c5b2b14337d8f9b265 for tablet baa240f65f74450e95664c3b5adb6da9. Status: Timed out: UpdateConsensus RPC to 127.4.126.1:38321 timed out after 1.000s (SENT). Retrying in the next heartbeat period. Already tried 1 times. I0819 06:34:41.295554 3486 tablet.cc:1390] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:41.296064 3486 tablet.cc:1445] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:41.296257 3486 tablet.cc:1447] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6156797873492078592 or (T in {6156797873492078592})}] I0819 06:34:41.297255 3611 tablet.cc:1390] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0819 06:34:41.297672 3611 tablet.cc:1445] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0819 06:34:41.297850 3611 tablet.cc:1447] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6156797873492078592 or (T in {6156797873492078592})}] I0819 06:34:41.382055 3755 consensus_peers.cc:212] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc -> Peer cd3b50a0af2548c5b2b14337d8f9b265 (127.4.126.1:38321): Could not obtain request from queue for peer: cd3b50a0af2548c5b2b14337d8f9b265. Status: Not found: Failed to read ops 3..29: Segment 1 which contained index 3 has been GCed I0819 06:34:41.382980 3755 raft_consensus.cc:770] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Attempting to remove follower cd3b50a0af2548c5b2b14337d8f9b265 from the Raft config. Reason: The logs necessary to catch up peer cd3b50a0af2548c5b2b14337d8f9b265 have been garbage collected. The follower will never be able to catch up (Not found: Failed to read ops 3..29: Segment 1 which contained index 3 has been GCed) I0819 06:34:41.383705 3755 consensus_peers.cc:418] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc -> Peer a331dc44a9014f3caa72b13d0954dd6c (127.4.126.3:50890): Closing peer: a331dc44a9014f3caa72b13d0954dd6c I0819 06:34:41.383900 3755 consensus_peers.cc:418] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc -> Peer cd3b50a0af2548c5b2b14337d8f9b265 (127.4.126.1:38321): Closing peer: cd3b50a0af2548c5b2b14337d8f9b265 I0819 06:34:41.384567 3755 consensus_queue.cc:169] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 1, Majority replicated index: 37, Committed index: 37, Last appended: 1.37, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "b50ee9ceeb65458495ef77d5d2dccbbc" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 51392 } } peers { permanent_uuid: "a331dc44a9014f3caa72b13d0954dd6c" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 50890 } } I0819 06:34:41.387442 3611 tablet.cc:1489] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Compaction successful on 30 rows (135937 bytes) I0819 06:34:41.389029 3611 maintenance_manager.cc:466] Time spent running CompactRowSetsOp(baa240f65f74450e95664c3b5adb6da9): real 0.389s user 0.336s sys 0.048s I0819 06:34:41.389240 3611 maintenance_manager.cc:472] P a331dc44a9014f3caa72b13d0954dd6c: CompactRowSetsOp(baa240f65f74450e95664c3b5adb6da9) metrics: {"cfile_cache_hit":10,"cfile_cache_hit_bytes":262394,"cfile_cache_miss":26,"cfile_cache_miss_bytes":264040,"cfile_init":5,"data dir 0.queue_time_us":1637,"data dir 0.run_cpu_time_us":449,"data dir 0.run_wall_time_us":2392,"delta_iterators_relevant":4,"lbm_read_time_us":4137,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":44,"lbm_write_time_us":1698,"lbm_writes_lt_1ms":23,"mutex_wait_us":14} I0819 06:34:41.391273 3611 maintenance_manager.cc:466] Time spent running UndoDeltaBlockGCOp(baa240f65f74450e95664c3b5adb6da9): real 0.001s user 0.000s sys 0.000s I0819 06:34:41.391468 3683 raft_consensus.cc:961] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c [term 1 FOLLOWER]: Refusing update from remote peer b50ee9ceeb65458495ef77d5d2dccbbc: Log matching property violated. Preceding OpId in replica: term: 1 index: 37. Preceding OpId from leader: term: 1 index: 38. (index mismatch) I0819 06:34:41.391470 3611 maintenance_manager.cc:472] P a331dc44a9014f3caa72b13d0954dd6c: UndoDeltaBlockGCOp(baa240f65f74450e95664c3b5adb6da9) metrics: {"cfile_init":1,"lbm_read_time_us":170,"lbm_reads_lt_1ms":4} I0819 06:34:41.393124 3765 consensus_queue.cc:726] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc [LEADER]: Connected to new peer: Peer: a331dc44a9014f3caa72b13d0954dd6c, Is new: false, Last received: 1.37, Next index: 38, Last known committed idx: 37, Last exchange result: ERROR, Needs tablet copy: false I0819 06:34:41.401433 3783 raft_consensus.cc:2339] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc [term 1 LEADER]: Committing config change with OpId 1.38: config changed from index -1 to 38, VOTER cd3b50a0af2548c5b2b14337d8f9b265 (127.4.126.1) evicted. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "b50ee9ceeb65458495ef77d5d2dccbbc" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 51392 } } peers { permanent_uuid: "a331dc44a9014f3caa72b13d0954dd6c" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 50890 } } } I0819 06:34:41.402542 3683 raft_consensus.cc:2339] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c [term 1 FOLLOWER]: Committing config change with OpId 1.38: config changed from index -1 to 38, VOTER cd3b50a0af2548c5b2b14337d8f9b265 (127.4.126.1) evicted. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "b50ee9ceeb65458495ef77d5d2dccbbc" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 51392 } } peers { permanent_uuid: "a331dc44a9014f3caa72b13d0954dd6c" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 50890 } } } I0819 06:34:41.409620 3308 catalog_manager.cc:2559] T baa240f65f74450e95664c3b5adb6da9 reported consensus state change: config changed from index -1 to 38, VOTER cd3b50a0af2548c5b2b14337d8f9b265 (127.4.126.1) evicted. New consensus state: current_term: 1 leader_uuid: "b50ee9ceeb65458495ef77d5d2dccbbc" committed_config { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "b50ee9ceeb65458495ef77d5d2dccbbc" member_type: VOTER last_known_addr { host: "127.4.126.2" port: 51392 } } peers { permanent_uuid: "a331dc44a9014f3caa72b13d0954dd6c" member_type: VOTER last_known_addr { host: "127.4.126.3" port: 50890 } } } I0819 06:34:41.410495 3308 catalog_manager.cc:3161] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet baa240f65f74450e95664c3b5adb6da9 on cd3b50a0af2548c5b2b14337d8f9b265 (127.4.126.1:38321) (TS cd3b50a0af2548c5b2b14337d8f9b265 not found in new config with opid_index 38) I0819 06:34:41.465457 3486 tablet.cc:1489] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Compaction successful on 29 rows (135937 bytes) I0819 06:34:41.466634 3486 maintenance_manager.cc:466] Time spent running CompactRowSetsOp(baa240f65f74450e95664c3b5adb6da9): real 0.458s user 0.320s sys 0.052s I0819 06:34:41.466799 3486 maintenance_manager.cc:472] P b50ee9ceeb65458495ef77d5d2dccbbc: CompactRowSetsOp(baa240f65f74450e95664c3b5adb6da9) metrics: {"cfile_cache_hit":10,"cfile_cache_hit_bytes":262394,"cfile_cache_miss":26,"cfile_cache_miss_bytes":264022,"cfile_init":5,"data dir 0.queue_time_us":893,"data dir 0.run_cpu_time_us":362,"data dir 0.run_wall_time_us":2281,"delta_iterators_relevant":4,"lbm_read_time_us":4100,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":45,"lbm_write_time_us":1700,"lbm_writes_lt_1ms":23} I0819 06:34:41.468325 3486 log_reader.cc:381] T baa240f65f74450e95664c3b5adb6da9: removed 1 log segments from log reader I0819 06:34:41.468516 3486 log.cc:868] T baa240f65f74450e95664c3b5adb6da9 P b50ee9ceeb65458495ef77d5d2dccbbc: Deleting log segment in path: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-1/wal/wals/baa240f65f74450e95664c3b5adb6da9/wal-000000004 (ops 23-29) I0819 06:34:41.468992 3486 maintenance_manager.cc:466] Time spent running LogGCOp(baa240f65f74450e95664c3b5adb6da9): real 0.001s user 0.000s sys 0.000s I0819 06:34:41.469143 3486 maintenance_manager.cc:472] P b50ee9ceeb65458495ef77d5d2dccbbc: LogGCOp(baa240f65f74450e95664c3b5adb6da9) metrics: {} I0819 06:34:41.470520 3486 maintenance_manager.cc:466] Time spent running UndoDeltaBlockGCOp(baa240f65f74450e95664c3b5adb6da9): real 0.001s user 0.000s sys 0.000s I0819 06:34:41.470670 3486 maintenance_manager.cc:472] P b50ee9ceeb65458495ef77d5d2dccbbc: UndoDeltaBlockGCOp(baa240f65f74450e95664c3b5adb6da9) metrics: {"cfile_init":1,"lbm_read_time_us":191,"lbm_reads_lt_1ms":4} I0819 06:34:41.492985 3611 log_reader.cc:381] T baa240f65f74450e95664c3b5adb6da9: removed 1 log segments from log reader I0819 06:34:41.493207 3611 log.cc:868] T baa240f65f74450e95664c3b5adb6da9 P a331dc44a9014f3caa72b13d0954dd6c: Deleting log segment in path: /tmp/run_tha_test479zHR/test-tmp/raft_consensus-itest.RaftConsensusITest.TestEvictAbandonedFollowers.1503124412820022-1150/raft_consensus-itest-cluster/ts-2/wal/wals/baa240f65f74450e95664c3b5adb6da9/wal-000000004 (ops 23-29) I0819 06:34:41.493737 3611 maintenance_manager.cc:466] Time spent running LogGCOp(baa240f65f74450e95664c3b5adb6da9): real 0.001s user 0.000s sys 0.000s I0819 06:34:41.493892 3611 maintenance_manager.cc:472] P a331dc44a9014f3caa72b13d0954dd6c: LogGCOp(baa240f65f74450e95664c3b5adb6da9) metrics: {} I0819 06:34:43.008363 1150 raft_consensus-itest.cc:762] Servers converged with original term 1 I0819 06:34:43.008522 1150 raft_consensus-itest.cc:766] Resuming cd3b50a0af2548c5b2b14337d8f9b265 I0819 06:34:43.011945 3413 tablet_service.cc:725] Processing DeleteTablet for tablet baa240f65f74450e95664c3b5adb6da9 with delete_type TABLET_DATA_TOMBSTONED (TS cd3b50a0af2548c5b2b14337d8f9b265 not found in new config with opid_index 38) from {username='slave'} at 127.0.0.1:58210 I0819 06:34:43.012403 3413 tablet_replica.cc:231] Initiating TabletReplica shutdown for tablet: baa240f65f74450e95664c3b5adb6da9 I0819 06:34:43.012629 3413 maintenance_manager.cc:209] P cd3b50a0af2548c5b2b14337d8f9b265: Unregistered op CompactRowSetsOp(baa240f65f74450e95664c3b5adb6da9) I0819 06:34:43.012823 3413 maintenance_manager.cc:209] P cd3b50a0af2548c5b2b14337d8f9b265: Unregistered op MinorDeltaCompactionOp(baa240f65f74450e95664c3b5adb6da9) I0819 06:34:43.012991 3413 maintenance_manager.cc:209] P cd3b50a0af2548c5b2b14337d8f9b265: Unregistered op MajorDeltaCompactionOp(baa240f65f74450e95664c3b5adb6da9) I0819 06:34:43.013142 3413 maintenance_manager.cc:209] P cd3b50a0af2548c5b2b14337d8f9b265: Unregistered op UndoDeltaBlockGCOp(baa240f65f74450e95664c3b5adb6da9) I0819 06:34:43.015836 1150 external_mini_cluster.cc:944] Killing /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver with pid 3343 I0819 06:34:43.014060 3756 raft_consensus.cc:413] T baa240f65f74450e95664c3b5adb6da9 P cd3b50a0af2548c5b2b14337d8f9b265 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b50ee9ceeb65458495ef77d5d2dccbbc) W0819 06:34:43.035923 3490 connection.cc:551] client connection to 127.4.126.1:38321 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:43.036010 3294 connection.cc:551] server connection from 127.4.126.1:45288 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:43.036006 3291 connection.cc:551] client connection to 127.4.126.1:38321 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:43.036538 3291 catalog_manager.cc:2897] TS cd3b50a0af2548c5b2b14337d8f9b265 (127.4.126.1:38321): Delete Tablet RPC failed for tablet baa240f65f74450e95664c3b5adb6da9: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0819 06:34:43.036746 3291 catalog_manager.cc:2932] Scheduling retry of baa240f65f74450e95664c3b5adb6da9 Delete Tablet RPC for TS=cd3b50a0af2548c5b2b14337d8f9b265 with a delay of 19ms (attempt = 1)... W0819 06:34:43.036898 3280 connection.cc:551] client connection to 127.4.126.1:38321 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:43.037619 3277 connection.cc:551] client connection to 127.4.126.1:38321 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0819 06:34:43.038009 1150 external_mini_cluster.cc:944] Killing /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver with pid 3480 I0819 06:34:43.057059 3291 catalog_manager.cc:3161] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet baa240f65f74450e95664c3b5adb6da9 on cd3b50a0af2548c5b2b14337d8f9b265 (127.4.126.1:38321) (TS cd3b50a0af2548c5b2b14337d8f9b265 not found in new config with opid_index 38) W0819 06:34:43.058854 3291 catalog_manager.cc:2897] TS cd3b50a0af2548c5b2b14337d8f9b265 (127.4.126.1:38321): Delete Tablet RPC failed for tablet baa240f65f74450e95664c3b5adb6da9: Network error: Client connection negotiation failed: client connection to 127.4.126.1:38321: connect: Connection refused (error 111) I0819 06:34:43.059072 3291 catalog_manager.cc:2932] Scheduling retry of baa240f65f74450e95664c3b5adb6da9 Delete Tablet RPC for TS=cd3b50a0af2548c5b2b14337d8f9b265 with a delay of 48ms (attempt = 2)... W0819 06:34:43.074852 3294 connection.cc:551] client connection to 127.4.126.2:51392 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:43.074903 3280 connection.cc:551] client connection to 127.4.126.2:51392 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:43.075711 3292 connection.cc:551] server connection from 127.4.126.2:45285 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:43.075861 3616 connection.cc:551] server connection from 127.4.126.2:54981 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:43.076782 3279 connection.cc:551] client connection to 127.4.126.2:51392 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0819 06:34:43.077409 1150 external_mini_cluster.cc:944] Killing /tmp/run_tha_test479zHR/build/tsan/bin/kudu-tserver with pid 3605 I0819 06:34:43.107769 3291 catalog_manager.cc:3161] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet baa240f65f74450e95664c3b5adb6da9 on cd3b50a0af2548c5b2b14337d8f9b265 (127.4.126.1:38321) (TS cd3b50a0af2548c5b2b14337d8f9b265 not found in new config with opid_index 38) W0819 06:34:43.109189 3291 catalog_manager.cc:2897] TS cd3b50a0af2548c5b2b14337d8f9b265 (127.4.126.1:38321): Delete Tablet RPC failed for tablet baa240f65f74450e95664c3b5adb6da9: Network error: Client connection negotiation failed: client connection to 127.4.126.1:38321: connect: Connection refused (error 111) I0819 06:34:43.109421 3291 catalog_manager.cc:2932] Scheduling retry of baa240f65f74450e95664c3b5adb6da9 Delete Tablet RPC for TS=cd3b50a0af2548c5b2b14337d8f9b265 with a delay of 111ms (attempt = 3)... W0819 06:34:43.116828 3292 connection.cc:551] client connection to 127.4.126.3:50890 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:43.116945 3280 connection.cc:551] client connection to 127.4.126.3:50890 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:43.117740 3278 connection.cc:551] client connection to 127.4.126.3:50890 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0819 06:34:43.117739 3294 connection.cc:551] server connection from 127.4.126.3:40807 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0819 06:34:43.118732 1150 external_mini_cluster.cc:944] Killing /tmp/run_tha_test479zHR/build/tsan/bin/kudu-master with pid 3281 W0819 06:34:43.148046 3280 connection.cc:551] client connection to 127.4.126.254:34857 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) [ OK ] RaftConsensusITest.TestEvictAbandonedFollowers (6460 ms) [----------] 5 tests from RaftConsensusITest (70247 ms total) [----------] Global test environment tear-down [==========] 5 tests from 1 test case ran. (70248 ms total) [ PASSED ] 5 tests. ThreadSanitizer: reported 3 warnings