[==========] Running 6 tests from 2 test cases. [----------] Global test environment set-up. [----------] 4 tests from TsTabletManagerITest [ RUN ] TsTabletManagerITest.TestReportNewLeaderOnLeaderChange WARNING: Logging before InitGoogleLogging() is written to STDERR W0916 07:55:32.425215 244 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.648s user 0.002s sys 0.000s W0916 07:55:32.425647 244 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.649s user 0.002s sys 0.000s I0916 07:55:32.430181 244 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.0.61.62:34869 I0916 07:55:32.451063 244 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 42016us I0916 07:55:32.451750 244 fs_manager.cc:263] Metadata directory not provided I0916 07:55:32.451907 244 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0916 07:55:32.452237 244 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0916 07:55:32.452384 244 server_base.cc:453] Attempting to create new FS layout instead I0916 07:55:32.456501 244 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568620531643032-244/minicluster-data/master-0-root/instance: uuid: "c1a56072d9734b5c943b74821e8a8dd3" format_stamp: "Formatted at 2019-09-16 07:55:32 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:32.468598 244 fs_manager.cc:503] Time spent creating directory manager: real 0.011s user 0.010s sys 0.001s I0916 07:55:32.469884 244 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0916 07:55:32.470484 244 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0916 07:55:32.480391 244 fs_manager.cc:419] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s I0916 07:55:32.480676 244 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568620531643032-244/minicluster-data/master-0-root uuid: "c1a56072d9734b5c943b74821e8a8dd3" format_stamp: "Formatted at 2019-09-16 07:55:32 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:32.481009 244 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568620531643032-244/minicluster-data/master-0-root metadata directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568620531643032-244/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568620531643032-244/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0916 07:55:32.540930 244 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0916 07:55:32.624500 244 rpc_server.cc:205] RPC server started. Bound to: 127.0.61.62:34869 I0916 07:55:32.628950 244 webserver.cc:374] Webserver started at http://127.0.61.62:37113/ using document root and password file I0916 07:55:32.632658 309 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0916 07:55:32.652189 309 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P c1a56072d9734b5c943b74821e8a8dd3: Bootstrap starting. I0916 07:55:32.655382 309 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P c1a56072d9734b5c943b74821e8a8dd3: No blocks or log segments found. Creating new log. I0916 07:55:32.657691 309 log.cc:525] T 00000000000000000000000000000000 P c1a56072d9734b5c943b74821e8a8dd3: Log is configured to *not* fsync() on all Append() calls I0916 07:55:32.660689 309 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P c1a56072d9734b5c943b74821e8a8dd3: No bootstrap required, opened a new log I0916 07:55:32.677119 309 raft_consensus.cc:340] T 00000000000000000000000000000000 P c1a56072d9734b5c943b74821e8a8dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1a56072d9734b5c943b74821e8a8dd3" member_type: VOTER } I0916 07:55:32.677565 309 raft_consensus.cc:366] T 00000000000000000000000000000000 P c1a56072d9734b5c943b74821e8a8dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0916 07:55:32.677721 309 raft_consensus.cc:705] T 00000000000000000000000000000000 P c1a56072d9734b5c943b74821e8a8dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1a56072d9734b5c943b74821e8a8dd3, State: Initialized, Role: FOLLOWER I0916 07:55:32.678328 309 consensus_queue.cc:229] T 00000000000000000000000000000000 P c1a56072d9734b5c943b74821e8a8dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1a56072d9734b5c943b74821e8a8dd3" member_type: VOTER } I0916 07:55:32.678731 309 raft_consensus.cc:380] T 00000000000000000000000000000000 P c1a56072d9734b5c943b74821e8a8dd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0916 07:55:32.678906 309 raft_consensus.cc:472] T 00000000000000000000000000000000 P c1a56072d9734b5c943b74821e8a8dd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0916 07:55:32.679136 309 raft_consensus.cc:2890] T 00000000000000000000000000000000 P c1a56072d9734b5c943b74821e8a8dd3 [term 0 FOLLOWER]: Advancing to term 1 I0916 07:55:32.683223 309 raft_consensus.cc:494] T 00000000000000000000000000000000 P c1a56072d9734b5c943b74821e8a8dd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1a56072d9734b5c943b74821e8a8dd3" member_type: VOTER } I0916 07:55:32.683813 309 leader_election.cc:310] T 00000000000000000000000000000000 P c1a56072d9734b5c943b74821e8a8dd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1a56072d9734b5c943b74821e8a8dd3; no voters: I0916 07:55:32.685667 311 raft_consensus.cc:2644] T 00000000000000000000000000000000 P c1a56072d9734b5c943b74821e8a8dd3 [term 1 FOLLOWER]: Leader election won for term 1 I0916 07:55:32.687543 309 leader_election.cc:296] T 00000000000000000000000000000000 P c1a56072d9734b5c943b74821e8a8dd3 [CANDIDATE]: Term 1 election: Requested vote from peers I0916 07:55:32.792496 311 raft_consensus.cc:667] T 00000000000000000000000000000000 P c1a56072d9734b5c943b74821e8a8dd3 [term 1 LEADER]: Becoming Leader. State: Replica: c1a56072d9734b5c943b74821e8a8dd3, State: Running, Role: LEADER I0916 07:55:32.793092 311 consensus_queue.cc:206] T 00000000000000000000000000000000 P c1a56072d9734b5c943b74821e8a8dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1a56072d9734b5c943b74821e8a8dd3" member_type: VOTER } I0916 07:55:32.798162 309 sys_catalog.cc:340] T 00000000000000000000000000000000 P c1a56072d9734b5c943b74821e8a8dd3 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "c1a56072d9734b5c943b74821e8a8dd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1a56072d9734b5c943b74821e8a8dd3" member_type: VOTER } } I0916 07:55:32.798120 311 sys_catalog.cc:340] T 00000000000000000000000000000000 P c1a56072d9734b5c943b74821e8a8dd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1a56072d9734b5c943b74821e8a8dd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1a56072d9734b5c943b74821e8a8dd3" member_type: VOTER } } I0916 07:55:32.798676 309 sys_catalog.cc:343] T 00000000000000000000000000000000 P c1a56072d9734b5c943b74821e8a8dd3 [sys.catalog]: This master's current role is: LEADER I0916 07:55:32.798683 311 sys_catalog.cc:343] T 00000000000000000000000000000000 P c1a56072d9734b5c943b74821e8a8dd3 [sys.catalog]: This master's current role is: LEADER I0916 07:55:32.798854 312 sys_catalog.cc:340] T 00000000000000000000000000000000 P c1a56072d9734b5c943b74821e8a8dd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1a56072d9734b5c943b74821e8a8dd3. Latest consensus state: current_term: 1 leader_uuid: "c1a56072d9734b5c943b74821e8a8dd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1a56072d9734b5c943b74821e8a8dd3" member_type: VOTER } } I0916 07:55:32.800595 312 sys_catalog.cc:343] T 00000000000000000000000000000000 P c1a56072d9734b5c943b74821e8a8dd3 [sys.catalog]: This master's current role is: LEADER I0916 07:55:32.801775 309 sys_catalog.cc:428] T 00000000000000000000000000000000 P c1a56072d9734b5c943b74821e8a8dd3 [sys.catalog]: configured and running, proceeding with master startup. I0916 07:55:32.805091 244 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 W0916 07:55:32.812949 316 catalog_manager.cc:1144] T 00000000000000000000000000000000 P c1a56072d9734b5c943b74821e8a8dd3: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0916 07:55:32.813144 316 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0916 07:55:32.813447 315 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0916 07:55:32.817265 315 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0916 07:55:32.878721 315 catalog_manager.cc:981] Generated new certificate authority record I0916 07:55:32.881194 315 catalog_manager.cc:1106] Loading token signing keys... I0916 07:55:32.894949 315 catalog_manager.cc:4435] T 00000000000000000000000000000000 P c1a56072d9734b5c943b74821e8a8dd3: Generated new TSK 0 I0916 07:55:32.949882 244 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 42016us I0916 07:55:32.950242 244 fs_manager.cc:263] Metadata directory not provided I0916 07:55:32.950386 244 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0916 07:55:32.950584 244 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0916 07:55:32.950686 244 server_base.cc:453] Attempting to create new FS layout instead I0916 07:55:32.951625 244 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568620531643032-244/minicluster-data/ts-0-root/instance: uuid: "141af11506424a92a1a70f34cbe1816c" format_stamp: "Formatted at 2019-09-16 07:55:32 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:32.961690 244 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.011s sys 0.000s I0916 07:55:32.963107 244 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0916 07:55:32.971326 244 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s I0916 07:55:32.971511 244 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568620531643032-244/minicluster-data/ts-0-root uuid: "141af11506424a92a1a70f34cbe1816c" format_stamp: "Formatted at 2019-09-16 07:55:32 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:32.971756 244 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568620531643032-244/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568620531643032-244/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568620531643032-244/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0916 07:55:33.003772 244 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0916 07:55:33.004035 244 ts_tablet_manager.cc:403] Registered 0 tablets I0916 07:55:33.131106 244 rpc_server.cc:205] RPC server started. Bound to: 127.0.61.1:41065 I0916 07:55:33.134320 244 webserver.cc:374] Webserver started at http://127.0.61.1:46383/ using document root and password file I0916 07:55:33.167524 384 heartbeater.cc:322] Connected to a master server at 127.0.61.62:34869 I0916 07:55:33.167863 384 heartbeater.cc:406] Registering TS with master... I0916 07:55:33.168403 384 heartbeater.cc:455] Master 127.0.61.62:34869 requested a full tablet report, sending... I0916 07:55:33.171252 244 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 42516us I0916 07:55:33.171643 244 fs_manager.cc:263] Metadata directory not provided I0916 07:55:33.171645 276 ts_manager.cc:161] Registered new tserver with Master: 141af11506424a92a1a70f34cbe1816c (127.0.61.1:41065) I0916 07:55:33.171876 244 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0916 07:55:33.172137 244 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0916 07:55:33.172287 244 server_base.cc:453] Attempting to create new FS layout instead I0916 07:55:33.173524 244 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568620531643032-244/minicluster-data/ts-1-root/instance: uuid: "0ec6c605f3f841deaf9a88d578a83da5" format_stamp: "Formatted at 2019-09-16 07:55:33 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:33.173902 276 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53192 I0916 07:55:33.187829 244 fs_manager.cc:503] Time spent creating directory manager: real 0.014s user 0.014s sys 0.000s I0916 07:55:33.189694 244 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0916 07:55:33.201385 244 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0916 07:55:33.201599 244 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568620531643032-244/minicluster-data/ts-1-root uuid: "0ec6c605f3f841deaf9a88d578a83da5" format_stamp: "Formatted at 2019-09-16 07:55:33 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:33.201825 244 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568620531643032-244/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568620531643032-244/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568620531643032-244/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0916 07:55:33.265404 244 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0916 07:55:33.265594 244 ts_tablet_manager.cc:403] Registered 0 tablets I0916 07:55:33.399739 244 rpc_server.cc:205] RPC server started. Bound to: 127.0.61.2:34717 I0916 07:55:33.401749 244 webserver.cc:374] Webserver started at http://127.0.61.2:32935/ using document root and password file I0916 07:55:33.421604 452 heartbeater.cc:322] Connected to a master server at 127.0.61.62:34869 I0916 07:55:33.421907 452 heartbeater.cc:406] Registering TS with master... I0916 07:55:33.422360 452 heartbeater.cc:455] Master 127.0.61.62:34869 requested a full tablet report, sending... I0916 07:55:33.423830 276 ts_manager.cc:161] Registered new tserver with Master: 0ec6c605f3f841deaf9a88d578a83da5 (127.0.61.2:34717) I0916 07:55:33.424141 244 internal_mini_cluster.cc:328] 2 TS(s) registered with all masters after 0.011654246s I0916 07:55:33.425546 276 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53194 I0916 07:55:33.462668 276 catalog_manager.cc:1440] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53196: name: "test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 2 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0916 07:55:33.476689 276 catalog_manager.cc:1605] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers. I0916 07:55:33.540736 419 tablet_service.cc:1022] Processing CreateTablet for tablet 2dfc43d2bb0645c89408e1c532f40f91 (table=test-table [id=3108cb7fe8ef41eb843663a02347d86f]), partition=RANGE (key) PARTITION UNBOUNDED I0916 07:55:33.541913 351 tablet_service.cc:1022] Processing CreateTablet for tablet 2dfc43d2bb0645c89408e1c532f40f91 (table=test-table [id=3108cb7fe8ef41eb843663a02347d86f]), partition=RANGE (key) PARTITION UNBOUNDED I0916 07:55:33.542115 419 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 2dfc43d2bb0645c89408e1c532f40f91. 1 dirs total, 0 dirs full, 0 dirs failed I0916 07:55:33.542786 351 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 2dfc43d2bb0645c89408e1c532f40f91. 1 dirs total, 0 dirs full, 0 dirs failed I0916 07:55:33.559235 469 tablet_bootstrap.cc:439] T 2dfc43d2bb0645c89408e1c532f40f91 P 141af11506424a92a1a70f34cbe1816c: Bootstrap starting. I0916 07:55:33.563071 469 tablet_bootstrap.cc:590] T 2dfc43d2bb0645c89408e1c532f40f91 P 141af11506424a92a1a70f34cbe1816c: No blocks or log segments found. Creating new log. I0916 07:55:33.564821 468 tablet_bootstrap.cc:439] T 2dfc43d2bb0645c89408e1c532f40f91 P 0ec6c605f3f841deaf9a88d578a83da5: Bootstrap starting. I0916 07:55:33.570392 468 tablet_bootstrap.cc:590] T 2dfc43d2bb0645c89408e1c532f40f91 P 0ec6c605f3f841deaf9a88d578a83da5: No blocks or log segments found. Creating new log. I0916 07:55:33.570633 469 tablet_bootstrap.cc:439] T 2dfc43d2bb0645c89408e1c532f40f91 P 141af11506424a92a1a70f34cbe1816c: No bootstrap required, opened a new log I0916 07:55:33.571077 469 ts_tablet_manager.cc:1085] T 2dfc43d2bb0645c89408e1c532f40f91 P 141af11506424a92a1a70f34cbe1816c: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.002s I0916 07:55:33.575855 469 raft_consensus.cc:340] T 2dfc43d2bb0645c89408e1c532f40f91 P 141af11506424a92a1a70f34cbe1816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec6c605f3f841deaf9a88d578a83da5" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 34717 } } peers { permanent_uuid: "141af11506424a92a1a70f34cbe1816c" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 41065 } } I0916 07:55:33.576580 469 raft_consensus.cc:705] T 2dfc43d2bb0645c89408e1c532f40f91 P 141af11506424a92a1a70f34cbe1816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 141af11506424a92a1a70f34cbe1816c, State: Initialized, Role: FOLLOWER I0916 07:55:33.576970 468 tablet_bootstrap.cc:439] T 2dfc43d2bb0645c89408e1c532f40f91 P 0ec6c605f3f841deaf9a88d578a83da5: No bootstrap required, opened a new log I0916 07:55:33.577425 468 ts_tablet_manager.cc:1085] T 2dfc43d2bb0645c89408e1c532f40f91 P 0ec6c605f3f841deaf9a88d578a83da5: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s I0916 07:55:33.577015 469 consensus_queue.cc:229] T 2dfc43d2bb0645c89408e1c532f40f91 P 141af11506424a92a1a70f34cbe1816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec6c605f3f841deaf9a88d578a83da5" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 34717 } } peers { permanent_uuid: "141af11506424a92a1a70f34cbe1816c" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 41065 } } I0916 07:55:33.580087 468 raft_consensus.cc:340] T 2dfc43d2bb0645c89408e1c532f40f91 P 0ec6c605f3f841deaf9a88d578a83da5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec6c605f3f841deaf9a88d578a83da5" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 34717 } } peers { permanent_uuid: "141af11506424a92a1a70f34cbe1816c" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 41065 } } I0916 07:55:33.580607 384 heartbeater.cc:447] Master 127.0.61.62:34869 was elected leader, sending a full tablet report... I0916 07:55:33.580725 468 raft_consensus.cc:705] T 2dfc43d2bb0645c89408e1c532f40f91 P 0ec6c605f3f841deaf9a88d578a83da5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec6c605f3f841deaf9a88d578a83da5, State: Initialized, Role: FOLLOWER I0916 07:55:33.581127 468 consensus_queue.cc:229] T 2dfc43d2bb0645c89408e1c532f40f91 P 0ec6c605f3f841deaf9a88d578a83da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec6c605f3f841deaf9a88d578a83da5" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 34717 } } peers { permanent_uuid: "141af11506424a92a1a70f34cbe1816c" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 41065 } } I0916 07:55:33.582896 469 ts_tablet_manager.cc:1113] T 2dfc43d2bb0645c89408e1c532f40f91 P 141af11506424a92a1a70f34cbe1816c: Time spent starting tablet: real 0.012s user 0.009s sys 0.000s I0916 07:55:33.587997 452 heartbeater.cc:447] Master 127.0.61.62:34869 was elected leader, sending a full tablet report... I0916 07:55:33.589280 468 ts_tablet_manager.cc:1113] T 2dfc43d2bb0645c89408e1c532f40f91 P 0ec6c605f3f841deaf9a88d578a83da5: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s I0916 07:55:33.620337 244 ts_tablet_manager-itest.cc:324] Electing peer 0... I0916 07:55:33.620599 244 raft_consensus.cc:405] T 2dfc43d2bb0645c89408e1c532f40f91 P 141af11506424a92a1a70f34cbe1816c [term 0 FOLLOWER]: Emulating election... I0916 07:55:33.620760 244 raft_consensus.cc:2890] T 2dfc43d2bb0645c89408e1c532f40f91 P 141af11506424a92a1a70f34cbe1816c [term 0 FOLLOWER]: Advancing to term 1 I0916 07:55:33.624192 244 raft_consensus.cc:667] T 2dfc43d2bb0645c89408e1c532f40f91 P 141af11506424a92a1a70f34cbe1816c [term 1 LEADER]: Becoming Leader. State: Replica: 141af11506424a92a1a70f34cbe1816c, State: Running, Role: LEADER W0916 07:55:33.624572 384 heartbeater.cc:571] Failed to heartbeat to 127.0.61.62:34869 (0 consecutive failures): IO error: failing all heartbeats for tests I0916 07:55:33.624677 244 consensus_queue.cc:206] T 2dfc43d2bb0645c89408e1c532f40f91 P 141af11506424a92a1a70f34cbe1816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec6c605f3f841deaf9a88d578a83da5" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 34717 } } peers { permanent_uuid: "141af11506424a92a1a70f34cbe1816c" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 41065 } } I0916 07:55:33.628371 244 ts_tablet_manager-itest.cc:327] Waiting for servers to agree... I0916 07:55:33.647435 244 cluster_itest_util.cc:259] Not converged past 1 yet: 1.1 0.0 W0916 07:55:33.647502 385 tablet.cc:1763] T 2dfc43d2bb0645c89408e1c532f40f91 P 141af11506424a92a1a70f34cbe1816c: Can't schedule compaction. Clean time has not been advanced past its initial value. I0916 07:55:33.751093 244 cluster_itest_util.cc:259] Not converged past 1 yet: 1.1 0.0 I0916 07:55:33.954999 244 cluster_itest_util.cc:259] Not converged past 1 yet: 1.1 0.0 I0916 07:55:34.180083 429 raft_consensus.cc:2890] T 2dfc43d2bb0645c89408e1c532f40f91 P 0ec6c605f3f841deaf9a88d578a83da5 [term 0 FOLLOWER]: Advancing to term 1 I0916 07:55:34.187196 476 consensus_queue.cc:984] T 2dfc43d2bb0645c89408e1c532f40f91 P 141af11506424a92a1a70f34cbe1816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec6c605f3f841deaf9a88d578a83da5" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 34717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0916 07:55:34.259729 244 ts_tablet_manager-itest.cc:324] Electing peer 1... I0916 07:55:34.259932 244 raft_consensus.cc:405] T 2dfc43d2bb0645c89408e1c532f40f91 P 0ec6c605f3f841deaf9a88d578a83da5 [term 1 FOLLOWER]: Emulating election... I0916 07:55:34.260126 244 raft_consensus.cc:2890] T 2dfc43d2bb0645c89408e1c532f40f91 P 0ec6c605f3f841deaf9a88d578a83da5 [term 1 FOLLOWER]: Advancing to term 2 I0916 07:55:34.264916 244 raft_consensus.cc:667] T 2dfc43d2bb0645c89408e1c532f40f91 P 0ec6c605f3f841deaf9a88d578a83da5 [term 2 LEADER]: Becoming Leader. State: Replica: 0ec6c605f3f841deaf9a88d578a83da5, State: Running, Role: LEADER I0916 07:55:34.265377 244 consensus_queue.cc:206] T 2dfc43d2bb0645c89408e1c532f40f91 P 0ec6c605f3f841deaf9a88d578a83da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec6c605f3f841deaf9a88d578a83da5" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 34717 } } peers { permanent_uuid: "141af11506424a92a1a70f34cbe1816c" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 41065 } } I0916 07:55:34.266947 244 ts_tablet_manager-itest.cc:327] Waiting for servers to agree... I0916 07:55:34.269965 244 cluster_itest_util.cc:259] Not converged past 2 yet: 1.1 2.2 I0916 07:55:34.373800 244 cluster_itest_util.cc:259] Not converged past 2 yet: 1.1 2.2 I0916 07:55:34.577385 244 cluster_itest_util.cc:259] Not converged past 2 yet: 1.1 2.2 I0916 07:55:34.706579 429 raft_consensus.cc:1149] T 2dfc43d2bb0645c89408e1c532f40f91 P 0ec6c605f3f841deaf9a88d578a83da5 [term 2 LEADER]: Rejecting Update request from peer 141af11506424a92a1a70f34cbe1816c for earlier term 1. Current term is 2. Ops: [] I0916 07:55:34.707618 481 consensus_queue.cc:995] T 2dfc43d2bb0645c89408e1c532f40f91 P 141af11506424a92a1a70f34cbe1816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ec6c605f3f841deaf9a88d578a83da5" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 34717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0916 07:55:34.708139 481 consensus_queue.cc:1162] T 2dfc43d2bb0645c89408e1c532f40f91 P 141af11506424a92a1a70f34cbe1816c [LEADER]: Peer 0ec6c605f3f841deaf9a88d578a83da5 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1 I0916 07:55:34.708434 476 raft_consensus.cc:2885] T 2dfc43d2bb0645c89408e1c532f40f91 P 141af11506424a92a1a70f34cbe1816c [term 1 LEADER]: Stepping down as leader of term 1 I0916 07:55:34.708644 476 raft_consensus.cc:705] T 2dfc43d2bb0645c89408e1c532f40f91 P 141af11506424a92a1a70f34cbe1816c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 141af11506424a92a1a70f34cbe1816c, State: Running, Role: LEADER I0916 07:55:34.708983 476 consensus_queue.cc:229] T 2dfc43d2bb0645c89408e1c532f40f91 P 141af11506424a92a1a70f34cbe1816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec6c605f3f841deaf9a88d578a83da5" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 34717 } } peers { permanent_uuid: "141af11506424a92a1a70f34cbe1816c" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 41065 } } I0916 07:55:34.709637 476 raft_consensus.cc:2890] T 2dfc43d2bb0645c89408e1c532f40f91 P 141af11506424a92a1a70f34cbe1816c [term 1 FOLLOWER]: Advancing to term 2 I0916 07:55:34.881304 244 cluster_itest_util.cc:259] Not converged past 2 yet: 1.1 2.2 I0916 07:55:34.890753 361 raft_consensus.cc:1184] T 2dfc43d2bb0645c89408e1c532f40f91 P 141af11506424a92a1a70f34cbe1816c [term 2 FOLLOWER]: Refusing update from remote peer 0ec6c605f3f841deaf9a88d578a83da5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch) I0916 07:55:34.891793 484 consensus_queue.cc:984] T 2dfc43d2bb0645c89408e1c532f40f91 P 0ec6c605f3f841deaf9a88d578a83da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "141af11506424a92a1a70f34cbe1816c" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 41065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0916 07:55:35.285733 244 ts_tablet_manager-itest.cc:324] Electing peer 1... I0916 07:55:35.285898 244 raft_consensus.cc:405] T 2dfc43d2bb0645c89408e1c532f40f91 P 0ec6c605f3f841deaf9a88d578a83da5 [term 2 LEADER]: Emulating election... I0916 07:55:35.286072 244 raft_consensus.cc:2885] T 2dfc43d2bb0645c89408e1c532f40f91 P 0ec6c605f3f841deaf9a88d578a83da5 [term 2 LEADER]: Stepping down as leader of term 2 I0916 07:55:35.286211 244 raft_consensus.cc:705] T 2dfc43d2bb0645c89408e1c532f40f91 P 0ec6c605f3f841deaf9a88d578a83da5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0ec6c605f3f841deaf9a88d578a83da5, State: Running, Role: LEADER I0916 07:55:35.286518 244 consensus_queue.cc:229] T 2dfc43d2bb0645c89408e1c532f40f91 P 0ec6c605f3f841deaf9a88d578a83da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec6c605f3f841deaf9a88d578a83da5" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 34717 } } peers { permanent_uuid: "141af11506424a92a1a70f34cbe1816c" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 41065 } } I0916 07:55:35.287104 244 raft_consensus.cc:2890] T 2dfc43d2bb0645c89408e1c532f40f91 P 0ec6c605f3f841deaf9a88d578a83da5 [term 2 FOLLOWER]: Advancing to term 3 I0916 07:55:35.290638 244 raft_consensus.cc:667] T 2dfc43d2bb0645c89408e1c532f40f91 P 0ec6c605f3f841deaf9a88d578a83da5 [term 3 LEADER]: Becoming Leader. State: Replica: 0ec6c605f3f841deaf9a88d578a83da5, State: Running, Role: LEADER I0916 07:55:35.291154 244 consensus_queue.cc:206] T 2dfc43d2bb0645c89408e1c532f40f91 P 0ec6c605f3f841deaf9a88d578a83da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec6c605f3f841deaf9a88d578a83da5" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 34717 } } peers { permanent_uuid: "141af11506424a92a1a70f34cbe1816c" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 41065 } } I0916 07:55:35.292510 244 ts_tablet_manager-itest.cc:327] Waiting for servers to agree... I0916 07:55:35.297112 244 cluster_itest_util.cc:259] Not converged past 3 yet: 2.2 3.3 I0916 07:55:35.401510 244 cluster_itest_util.cc:259] Not converged past 3 yet: 2.2 3.3 I0916 07:55:35.605499 244 cluster_itest_util.cc:259] Not converged past 3 yet: 2.2 3.3 I0916 07:55:35.784523 361 raft_consensus.cc:2890] T 2dfc43d2bb0645c89408e1c532f40f91 P 141af11506424a92a1a70f34cbe1816c [term 2 FOLLOWER]: Advancing to term 3 I0916 07:55:35.788770 361 raft_consensus.cc:1184] T 2dfc43d2bb0645c89408e1c532f40f91 P 141af11506424a92a1a70f34cbe1816c [term 3 FOLLOWER]: Refusing update from remote peer 0ec6c605f3f841deaf9a88d578a83da5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch) I0916 07:55:35.790040 484 consensus_queue.cc:984] T 2dfc43d2bb0645c89408e1c532f40f91 P 0ec6c605f3f841deaf9a88d578a83da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "141af11506424a92a1a70f34cbe1816c" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 41065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0916 07:55:35.915702 244 tablet_server.cc:159] TabletServer@127.0.61.1:41065 shutting down... I0916 07:55:35.930315 244 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0916 07:55:35.930595 244 tablet_replica.cc:273] T 2dfc43d2bb0645c89408e1c532f40f91 P 141af11506424a92a1a70f34cbe1816c: stopping tablet replica I0916 07:55:35.930958 244 raft_consensus.cc:2147] T 2dfc43d2bb0645c89408e1c532f40f91 P 141af11506424a92a1a70f34cbe1816c [term 3 FOLLOWER]: Raft consensus shutting down. I0916 07:55:35.931246 244 raft_consensus.cc:2174] T 2dfc43d2bb0645c89408e1c532f40f91 P 141af11506424a92a1a70f34cbe1816c [term 3 FOLLOWER]: Raft consensus is shut down! I0916 07:55:36.011404 244 tablet_server.cc:173] TabletServer@127.0.61.1:41065 shutdown complete. I0916 07:55:36.017793 244 tablet_server.cc:159] TabletServer@127.0.61.2:34717 shutting down... I0916 07:55:36.031610 244 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0916 07:55:36.031895 244 tablet_replica.cc:273] T 2dfc43d2bb0645c89408e1c532f40f91 P 0ec6c605f3f841deaf9a88d578a83da5: stopping tablet replica I0916 07:55:36.032173 244 raft_consensus.cc:2147] T 2dfc43d2bb0645c89408e1c532f40f91 P 0ec6c605f3f841deaf9a88d578a83da5 [term 3 LEADER]: Raft consensus shutting down. I0916 07:55:36.032626 244 raft_consensus.cc:2174] T 2dfc43d2bb0645c89408e1c532f40f91 P 0ec6c605f3f841deaf9a88d578a83da5 [term 3 FOLLOWER]: Raft consensus is shut down! I0916 07:55:36.058912 244 tablet_server.cc:173] TabletServer@127.0.61.2:34717 shutdown complete. I0916 07:55:36.064622 244 master.cc:247] Master@127.0.61.62:34869 shutting down... I0916 07:55:36.120921 244 raft_consensus.cc:2147] T 00000000000000000000000000000000 P c1a56072d9734b5c943b74821e8a8dd3 [term 1 LEADER]: Raft consensus shutting down. I0916 07:55:36.121301 244 raft_consensus.cc:2174] T 00000000000000000000000000000000 P c1a56072d9734b5c943b74821e8a8dd3 [term 1 FOLLOWER]: Raft consensus is shut down! I0916 07:55:36.121495 244 tablet_replica.cc:273] T 00000000000000000000000000000000 P c1a56072d9734b5c943b74821e8a8dd3: stopping tablet replica I0916 07:55:36.137439 244 master.cc:258] Master@127.0.61.62:34869 shutdown complete. [ OK ] TsTabletManagerITest.TestReportNewLeaderOnLeaderChange (4419 ms) [ RUN ] TsTabletManagerITest.ReportOnReplicaHealthStatus I0916 07:55:36.157083 244 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.0.61.62:39659 I0916 07:55:36.168509 244 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 44016us I0916 07:55:36.168874 244 fs_manager.cc:263] Metadata directory not provided I0916 07:55:36.168995 244 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0916 07:55:36.169160 244 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0916 07:55:36.169265 244 server_base.cc:453] Attempting to create new FS layout instead I0916 07:55:36.170120 244 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568620531643032-244/minicluster-data/master-0-root/instance: uuid: "fd268f40e6b840f3a8b2334c84e40210" format_stamp: "Formatted at 2019-09-16 07:55:36 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:36.177829 244 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s I0916 07:55:36.179102 244 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0916 07:55:36.185320 244 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0916 07:55:36.185482 244 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568620531643032-244/minicluster-data/master-0-root uuid: "fd268f40e6b840f3a8b2334c84e40210" format_stamp: "Formatted at 2019-09-16 07:55:36 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:36.185704 244 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568620531643032-244/minicluster-data/master-0-root metadata directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568620531643032-244/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568620531643032-244/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0916 07:55:36.265365 244 rpc_server.cc:205] RPC server started. Bound to: 127.0.61.62:39659 I0916 07:55:36.266940 244 webserver.cc:374] Webserver started at http://127.0.61.62:36157/ using document root and password file I0916 07:55:36.268580 552 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0916 07:55:36.275315 552 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P fd268f40e6b840f3a8b2334c84e40210: Bootstrap starting. I0916 07:55:36.277418 552 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P fd268f40e6b840f3a8b2334c84e40210: No blocks or log segments found. Creating new log. I0916 07:55:36.280099 552 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P fd268f40e6b840f3a8b2334c84e40210: No bootstrap required, opened a new log I0916 07:55:36.281445 552 raft_consensus.cc:340] T 00000000000000000000000000000000 P fd268f40e6b840f3a8b2334c84e40210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd268f40e6b840f3a8b2334c84e40210" member_type: VOTER } I0916 07:55:36.281769 552 raft_consensus.cc:366] T 00000000000000000000000000000000 P fd268f40e6b840f3a8b2334c84e40210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0916 07:55:36.281932 552 raft_consensus.cc:705] T 00000000000000000000000000000000 P fd268f40e6b840f3a8b2334c84e40210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd268f40e6b840f3a8b2334c84e40210, State: Initialized, Role: FOLLOWER I0916 07:55:36.282346 552 consensus_queue.cc:229] T 00000000000000000000000000000000 P fd268f40e6b840f3a8b2334c84e40210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd268f40e6b840f3a8b2334c84e40210" member_type: VOTER } I0916 07:55:36.282714 552 raft_consensus.cc:380] T 00000000000000000000000000000000 P fd268f40e6b840f3a8b2334c84e40210 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0916 07:55:36.282868 552 raft_consensus.cc:472] T 00000000000000000000000000000000 P fd268f40e6b840f3a8b2334c84e40210 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0916 07:55:36.283074 552 raft_consensus.cc:2890] T 00000000000000000000000000000000 P fd268f40e6b840f3a8b2334c84e40210 [term 0 FOLLOWER]: Advancing to term 1 I0916 07:55:36.286322 552 raft_consensus.cc:494] T 00000000000000000000000000000000 P fd268f40e6b840f3a8b2334c84e40210 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd268f40e6b840f3a8b2334c84e40210" member_type: VOTER } I0916 07:55:36.286721 552 leader_election.cc:310] T 00000000000000000000000000000000 P fd268f40e6b840f3a8b2334c84e40210 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd268f40e6b840f3a8b2334c84e40210; no voters: I0916 07:55:36.287767 552 leader_election.cc:296] T 00000000000000000000000000000000 P fd268f40e6b840f3a8b2334c84e40210 [CANDIDATE]: Term 1 election: Requested vote from peers I0916 07:55:36.287984 554 raft_consensus.cc:2644] T 00000000000000000000000000000000 P fd268f40e6b840f3a8b2334c84e40210 [term 1 FOLLOWER]: Leader election won for term 1 I0916 07:55:36.289517 554 raft_consensus.cc:667] T 00000000000000000000000000000000 P fd268f40e6b840f3a8b2334c84e40210 [term 1 LEADER]: Becoming Leader. State: Replica: fd268f40e6b840f3a8b2334c84e40210, State: Running, Role: LEADER I0916 07:55:36.289894 554 consensus_queue.cc:206] T 00000000000000000000000000000000 P fd268f40e6b840f3a8b2334c84e40210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd268f40e6b840f3a8b2334c84e40210" member_type: VOTER } I0916 07:55:36.292253 552 sys_catalog.cc:340] T 00000000000000000000000000000000 P fd268f40e6b840f3a8b2334c84e40210 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "fd268f40e6b840f3a8b2334c84e40210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd268f40e6b840f3a8b2334c84e40210" member_type: VOTER } } I0916 07:55:36.292429 555 sys_catalog.cc:340] T 00000000000000000000000000000000 P fd268f40e6b840f3a8b2334c84e40210 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd268f40e6b840f3a8b2334c84e40210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd268f40e6b840f3a8b2334c84e40210" member_type: VOTER } } I0916 07:55:36.292294 556 sys_catalog.cc:340] T 00000000000000000000000000000000 P fd268f40e6b840f3a8b2334c84e40210 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd268f40e6b840f3a8b2334c84e40210. Latest consensus state: current_term: 1 leader_uuid: "fd268f40e6b840f3a8b2334c84e40210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd268f40e6b840f3a8b2334c84e40210" member_type: VOTER } } I0916 07:55:36.292822 552 sys_catalog.cc:343] T 00000000000000000000000000000000 P fd268f40e6b840f3a8b2334c84e40210 [sys.catalog]: This master's current role is: LEADER I0916 07:55:36.292908 555 sys_catalog.cc:343] T 00000000000000000000000000000000 P fd268f40e6b840f3a8b2334c84e40210 [sys.catalog]: This master's current role is: LEADER I0916 07:55:36.292917 556 sys_catalog.cc:343] T 00000000000000000000000000000000 P fd268f40e6b840f3a8b2334c84e40210 [sys.catalog]: This master's current role is: LEADER I0916 07:55:36.295085 552 sys_catalog.cc:428] T 00000000000000000000000000000000 P fd268f40e6b840f3a8b2334c84e40210 [sys.catalog]: configured and running, proceeding with master startup. I0916 07:55:36.295347 558 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0916 07:55:36.299446 558 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0916 07:55:36.305225 244 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0916 07:55:36.339293 558 catalog_manager.cc:981] Generated new certificate authority record I0916 07:55:36.340728 558 catalog_manager.cc:1106] Loading token signing keys... I0916 07:55:36.349833 558 catalog_manager.cc:4435] T 00000000000000000000000000000000 P fd268f40e6b840f3a8b2334c84e40210: Generated new TSK 0 I0916 07:55:36.381983 244 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 44016us I0916 07:55:36.382356 244 fs_manager.cc:263] Metadata directory not provided I0916 07:55:36.382498 244 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0916 07:55:36.382683 244 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0916 07:55:36.382786 244 server_base.cc:453] Attempting to create new FS layout instead I0916 07:55:36.383689 244 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568620531643032-244/minicluster-data/ts-0-root/instance: uuid: "b90a2dd4b68e4342b3a38b0d10ccff9a" format_stamp: "Formatted at 2019-09-16 07:55:36 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:36.391541 244 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.009s sys 0.001s I0916 07:55:36.393003 244 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0916 07:55:36.399134 244 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0916 07:55:36.399332 244 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568620531643032-244/minicluster-data/ts-0-root uuid: "b90a2dd4b68e4342b3a38b0d10ccff9a" format_stamp: "Formatted at 2019-09-16 07:55:36 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:36.399577 244 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568620531643032-244/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568620531643032-244/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568620531643032-244/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0916 07:55:36.461728 244 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0916 07:55:36.461890 244 ts_tablet_manager.cc:403] Registered 0 tablets I0916 07:55:36.502987 244 rpc_server.cc:205] RPC server started. Bound to: 127.0.61.1:43745 I0916 07:55:36.504773 244 webserver.cc:374] Webserver started at http://127.0.61.1:36025/ using document root and password file I0916 07:55:36.518308 626 heartbeater.cc:322] Connected to a master server at 127.0.61.62:39659 I0916 07:55:36.518582 626 heartbeater.cc:406] Registering TS with master... I0916 07:55:36.518990 626 heartbeater.cc:455] Master 127.0.61.62:39659 requested a full tablet report, sending... I0916 07:55:36.520630 519 ts_manager.cc:161] Registered new tserver with Master: b90a2dd4b68e4342b3a38b0d10ccff9a (127.0.61.1:43745) I0916 07:55:36.522912 519 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43948 I0916 07:55:36.523011 244 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 44016us I0916 07:55:36.523412 244 fs_manager.cc:263] Metadata directory not provided I0916 07:55:36.523604 244 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0916 07:55:36.523838 244 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0916 07:55:36.523980 244 server_base.cc:453] Attempting to create new FS layout instead I0916 07:55:36.524914 244 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568620531643032-244/minicluster-data/ts-1-root/instance: uuid: "d1bfd566b5df4d4db14378793b6a2f65" format_stamp: "Formatted at 2019-09-16 07:55:36 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:36.532907 244 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s I0916 07:55:36.534297 244 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0916 07:55:36.540555 244 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0916 07:55:36.540715 244 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568620531643032-244/minicluster-data/ts-1-root uuid: "d1bfd566b5df4d4db14378793b6a2f65" format_stamp: "Formatted at 2019-09-16 07:55:36 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:36.540962 244 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568620531643032-244/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568620531643032-244/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568620531643032-244/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0916 07:55:36.593788 244 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0916 07:55:36.593966 244 ts_tablet_manager.cc:403] Registered 0 tablets I0916 07:55:36.641057 244 rpc_server.cc:205] RPC server started. Bound to: 127.0.61.2:42715 I0916 07:55:36.642999 244 webserver.cc:374] Webserver started at http://127.0.61.2:34665/ using document root and password file I0916 07:55:36.655325 694 heartbeater.cc:322] Connected to a master server at 127.0.61.62:39659 I0916 07:55:36.655634 694 heartbeater.cc:406] Registering TS with master... I0916 07:55:36.656141 694 heartbeater.cc:455] Master 127.0.61.62:39659 requested a full tablet report, sending... I0916 07:55:36.657773 519 ts_manager.cc:161] Registered new tserver with Master: d1bfd566b5df4d4db14378793b6a2f65 (127.0.61.2:42715) I0916 07:55:36.659447 519 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43950 I0916 07:55:36.661705 244 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 44016us I0916 07:55:36.662117 244 fs_manager.cc:263] Metadata directory not provided I0916 07:55:36.662295 244 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0916 07:55:36.662523 244 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0916 07:55:36.662628 244 server_base.cc:453] Attempting to create new FS layout instead I0916 07:55:36.663431 244 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568620531643032-244/minicluster-data/ts-2-root/instance: uuid: "e4c9a84955dd4a63aea4a185fd53f118" format_stamp: "Formatted at 2019-09-16 07:55:36 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:36.672039 244 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.010s sys 0.001s I0916 07:55:36.673323 244 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0916 07:55:36.680125 244 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0916 07:55:36.680275 244 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568620531643032-244/minicluster-data/ts-2-root uuid: "e4c9a84955dd4a63aea4a185fd53f118" format_stamp: "Formatted at 2019-09-16 07:55:36 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:36.680524 244 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568620531643032-244/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568620531643032-244/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568620531643032-244/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0916 07:55:36.747285 244 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0916 07:55:36.747449 244 ts_tablet_manager.cc:403] Registered 0 tablets I0916 07:55:36.832864 244 rpc_server.cc:205] RPC server started. Bound to: 127.0.61.3:46265 I0916 07:55:36.836624 244 webserver.cc:374] Webserver started at http://127.0.61.3:35055/ using document root and password file I0916 07:55:36.851428 761 heartbeater.cc:322] Connected to a master server at 127.0.61.62:39659 I0916 07:55:36.851701 761 heartbeater.cc:406] Registering TS with master... I0916 07:55:36.852084 761 heartbeater.cc:455] Master 127.0.61.62:39659 requested a full tablet report, sending... I0916 07:55:36.853637 519 ts_manager.cc:161] Registered new tserver with Master: e4c9a84955dd4a63aea4a185fd53f118 (127.0.61.3:46265) I0916 07:55:36.854311 244 internal_mini_cluster.cc:328] 3 TS(s) registered with all masters after 0.01368708s I0916 07:55:36.855331 519 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43952 I0916 07:55:36.875221 519 catalog_manager.cc:1440] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43954: name: "test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { } } W0916 07:55:36.876015 519 catalog_manager.cc:1605] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0916 07:55:36.932106 728 tablet_service.cc:1022] Processing CreateTablet for tablet 2d457d87556c49d1a61e3a4f63de087d (table=test-table [id=17a5ded3f2a744edbd500672ec8caa2d]), partition= I0916 07:55:36.932788 728 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 2d457d87556c49d1a61e3a4f63de087d. 1 dirs total, 0 dirs full, 0 dirs failed I0916 07:55:36.938982 593 tablet_service.cc:1022] Processing CreateTablet for tablet 2d457d87556c49d1a61e3a4f63de087d (table=test-table [id=17a5ded3f2a744edbd500672ec8caa2d]), partition= I0916 07:55:36.939635 593 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 2d457d87556c49d1a61e3a4f63de087d. 1 dirs total, 0 dirs full, 0 dirs failed I0916 07:55:36.945385 661 tablet_service.cc:1022] Processing CreateTablet for tablet 2d457d87556c49d1a61e3a4f63de087d (table=test-table [id=17a5ded3f2a744edbd500672ec8caa2d]), partition= I0916 07:55:36.946197 661 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 2d457d87556c49d1a61e3a4f63de087d. 1 dirs total, 0 dirs full, 0 dirs failed I0916 07:55:36.948855 779 tablet_bootstrap.cc:439] T 2d457d87556c49d1a61e3a4f63de087d P e4c9a84955dd4a63aea4a185fd53f118: Bootstrap starting. I0916 07:55:36.953991 779 tablet_bootstrap.cc:590] T 2d457d87556c49d1a61e3a4f63de087d P e4c9a84955dd4a63aea4a185fd53f118: No blocks or log segments found. Creating new log. I0916 07:55:36.960395 780 tablet_bootstrap.cc:439] T 2d457d87556c49d1a61e3a4f63de087d P b90a2dd4b68e4342b3a38b0d10ccff9a: Bootstrap starting. I0916 07:55:36.961428 779 tablet_bootstrap.cc:439] T 2d457d87556c49d1a61e3a4f63de087d P e4c9a84955dd4a63aea4a185fd53f118: No bootstrap required, opened a new log I0916 07:55:36.961903 779 ts_tablet_manager.cc:1085] T 2d457d87556c49d1a61e3a4f63de087d P e4c9a84955dd4a63aea4a185fd53f118: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s I0916 07:55:36.964994 782 tablet_bootstrap.cc:439] T 2d457d87556c49d1a61e3a4f63de087d P d1bfd566b5df4d4db14378793b6a2f65: Bootstrap starting. I0916 07:55:36.966260 779 raft_consensus.cc:340] T 2d457d87556c49d1a61e3a4f63de087d P e4c9a84955dd4a63aea4a185fd53f118 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c9a84955dd4a63aea4a185fd53f118" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 46265 } } peers { permanent_uuid: "b90a2dd4b68e4342b3a38b0d10ccff9a" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 43745 } } peers { permanent_uuid: "d1bfd566b5df4d4db14378793b6a2f65" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 42715 } } I0916 07:55:36.966708 780 tablet_bootstrap.cc:590] T 2d457d87556c49d1a61e3a4f63de087d P b90a2dd4b68e4342b3a38b0d10ccff9a: No blocks or log segments found. Creating new log. I0916 07:55:36.966830 779 raft_consensus.cc:366] T 2d457d87556c49d1a61e3a4f63de087d P e4c9a84955dd4a63aea4a185fd53f118 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0916 07:55:36.967063 779 raft_consensus.cc:705] T 2d457d87556c49d1a61e3a4f63de087d P e4c9a84955dd4a63aea4a185fd53f118 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c9a84955dd4a63aea4a185fd53f118, State: Initialized, Role: FOLLOWER I0916 07:55:36.967602 779 consensus_queue.cc:229] T 2d457d87556c49d1a61e3a4f63de087d P e4c9a84955dd4a63aea4a185fd53f118 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c9a84955dd4a63aea4a185fd53f118" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 46265 } } peers { permanent_uuid: "b90a2dd4b68e4342b3a38b0d10ccff9a" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 43745 } } peers { permanent_uuid: "d1bfd566b5df4d4db14378793b6a2f65" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 42715 } } I0916 07:55:36.972782 761 heartbeater.cc:447] Master 127.0.61.62:39659 was elected leader, sending a full tablet report... I0916 07:55:36.973069 782 tablet_bootstrap.cc:590] T 2d457d87556c49d1a61e3a4f63de087d P d1bfd566b5df4d4db14378793b6a2f65: No blocks or log segments found. Creating new log. I0916 07:55:36.973275 779 ts_tablet_manager.cc:1113] T 2d457d87556c49d1a61e3a4f63de087d P e4c9a84955dd4a63aea4a185fd53f118: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s I0916 07:55:36.978744 780 tablet_bootstrap.cc:439] T 2d457d87556c49d1a61e3a4f63de087d P b90a2dd4b68e4342b3a38b0d10ccff9a: No bootstrap required, opened a new log I0916 07:55:36.979140 780 ts_tablet_manager.cc:1085] T 2d457d87556c49d1a61e3a4f63de087d P b90a2dd4b68e4342b3a38b0d10ccff9a: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.006s I0916 07:55:36.981456 780 raft_consensus.cc:340] T 2d457d87556c49d1a61e3a4f63de087d P b90a2dd4b68e4342b3a38b0d10ccff9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c9a84955dd4a63aea4a185fd53f118" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 46265 } } peers { permanent_uuid: "b90a2dd4b68e4342b3a38b0d10ccff9a" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 43745 } } peers { permanent_uuid: "d1bfd566b5df4d4db14378793b6a2f65" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 42715 } } I0916 07:55:36.982133 780 raft_consensus.cc:366] T 2d457d87556c49d1a61e3a4f63de087d P b90a2dd4b68e4342b3a38b0d10ccff9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0916 07:55:36.982331 780 raft_consensus.cc:705] T 2d457d87556c49d1a61e3a4f63de087d P b90a2dd4b68e4342b3a38b0d10ccff9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b90a2dd4b68e4342b3a38b0d10ccff9a, State: Initialized, Role: FOLLOWER I0916 07:55:36.982821 780 consensus_queue.cc:229] T 2d457d87556c49d1a61e3a4f63de087d P b90a2dd4b68e4342b3a38b0d10ccff9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c9a84955dd4a63aea4a185fd53f118" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 46265 } } peers { permanent_uuid: "b90a2dd4b68e4342b3a38b0d10ccff9a" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 43745 } } peers { permanent_uuid: "d1bfd566b5df4d4db14378793b6a2f65" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 42715 } } I0916 07:55:36.986061 782 tablet_bootstrap.cc:439] T 2d457d87556c49d1a61e3a4f63de087d P d1bfd566b5df4d4db14378793b6a2f65: No bootstrap required, opened a new log I0916 07:55:36.986389 626 heartbeater.cc:447] Master 127.0.61.62:39659 was elected leader, sending a full tablet report... I0916 07:55:36.986621 782 ts_tablet_manager.cc:1085] T 2d457d87556c49d1a61e3a4f63de087d P d1bfd566b5df4d4db14378793b6a2f65: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.004s I0916 07:55:36.987711 780 ts_tablet_manager.cc:1113] T 2d457d87556c49d1a61e3a4f63de087d P b90a2dd4b68e4342b3a38b0d10ccff9a: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s I0916 07:55:36.994336 783 raft_consensus.cc:472] T 2d457d87556c49d1a61e3a4f63de087d P e4c9a84955dd4a63aea4a185fd53f118 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0916 07:55:36.993985 782 raft_consensus.cc:340] T 2d457d87556c49d1a61e3a4f63de087d P d1bfd566b5df4d4db14378793b6a2f65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c9a84955dd4a63aea4a185fd53f118" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 46265 } } peers { permanent_uuid: "b90a2dd4b68e4342b3a38b0d10ccff9a" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 43745 } } peers { permanent_uuid: "d1bfd566b5df4d4db14378793b6a2f65" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 42715 } } I0916 07:55:36.994845 782 raft_consensus.cc:366] T 2d457d87556c49d1a61e3a4f63de087d P d1bfd566b5df4d4db14378793b6a2f65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0916 07:55:36.995090 782 raft_consensus.cc:705] T 2d457d87556c49d1a61e3a4f63de087d P d1bfd566b5df4d4db14378793b6a2f65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1bfd566b5df4d4db14378793b6a2f65, State: Initialized, Role: FOLLOWER I0916 07:55:36.994760 783 raft_consensus.cc:494] T 2d457d87556c49d1a61e3a4f63de087d P e4c9a84955dd4a63aea4a185fd53f118 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c9a84955dd4a63aea4a185fd53f118" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 46265 } } peers { permanent_uuid: "b90a2dd4b68e4342b3a38b0d10ccff9a" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 43745 } } peers { permanent_uuid: "d1bfd566b5df4d4db14378793b6a2f65" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 42715 } } I0916 07:55:36.995923 782 consensus_queue.cc:229] T 2d457d87556c49d1a61e3a4f63de087d P d1bfd566b5df4d4db14378793b6a2f65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c9a84955dd4a63aea4a185fd53f118" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 46265 } } peers { permanent_uuid: "b90a2dd4b68e4342b3a38b0d10ccff9a" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 43745 } } peers { permanent_uuid: "d1bfd566b5df4d4db14378793b6a2f65" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 42715 } } I0916 07:55:36.999539 783 leader_election.cc:296] T 2d457d87556c49d1a61e3a4f63de087d P e4c9a84955dd4a63aea4a185fd53f118 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b90a2dd4b68e4342b3a38b0d10ccff9a (127.0.61.1:43745), d1bfd566b5df4d4db14378793b6a2f65 (127.0.61.2:42715) I0916 07:55:37.004904 694 heartbeater.cc:447] Master 127.0.61.62:39659 was elected leader, sending a full tablet report... I0916 07:55:37.006485 782 ts_tablet_manager.cc:1113] T 2d457d87556c49d1a61e3a4f63de087d P d1bfd566b5df4d4db14378793b6a2f65: Time spent starting tablet: real 0.020s user 0.008s sys 0.001s I0916 07:55:37.018544 671 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "2d457d87556c49d1a61e3a4f63de087d" candidate_uuid: "e4c9a84955dd4a63aea4a185fd53f118" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1bfd566b5df4d4db14378793b6a2f65" is_pre_election: true I0916 07:55:37.019366 671 raft_consensus.cc:2350] T 2d457d87556c49d1a61e3a4f63de087d P d1bfd566b5df4d4db14378793b6a2f65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4c9a84955dd4a63aea4a185fd53f118 in term 0. I0916 07:55:37.020570 701 leader_election.cc:310] T 2d457d87556c49d1a61e3a4f63de087d P e4c9a84955dd4a63aea4a185fd53f118 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1bfd566b5df4d4db14378793b6a2f65, e4c9a84955dd4a63aea4a185fd53f118; no voters: I0916 07:55:37.021240 783 raft_consensus.cc:2644] T 2d457d87556c49d1a61e3a4f63de087d P e4c9a84955dd4a63aea4a185fd53f118 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0916 07:55:37.021525 783 raft_consensus.cc:472] T 2d457d87556c49d1a61e3a4f63de087d P e4c9a84955dd4a63aea4a185fd53f118 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0916 07:55:37.021844 783 raft_consensus.cc:2890] T 2d457d87556c49d1a61e3a4f63de087d P e4c9a84955dd4a63aea4a185fd53f118 [term 0 FOLLOWER]: Advancing to term 1 I0916 07:55:37.025501 603 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "2d457d87556c49d1a61e3a4f63de087d" candidate_uuid: "e4c9a84955dd4a63aea4a185fd53f118" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b90a2dd4b68e4342b3a38b0d10ccff9a" is_pre_election: true I0916 07:55:37.026190 603 raft_consensus.cc:2350] T 2d457d87556c49d1a61e3a4f63de087d P b90a2dd4b68e4342b3a38b0d10ccff9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4c9a84955dd4a63aea4a185fd53f118 in term 0. I0916 07:55:37.027951 783 raft_consensus.cc:494] T 2d457d87556c49d1a61e3a4f63de087d P e4c9a84955dd4a63aea4a185fd53f118 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c9a84955dd4a63aea4a185fd53f118" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 46265 } } peers { permanent_uuid: "b90a2dd4b68e4342b3a38b0d10ccff9a" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 43745 } } peers { permanent_uuid: "d1bfd566b5df4d4db14378793b6a2f65" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 42715 } } I0916 07:55:37.029510 783 leader_election.cc:296] T 2d457d87556c49d1a61e3a4f63de087d P e4c9a84955dd4a63aea4a185fd53f118 [CANDIDATE]: Term 1 election: Requested vote from peers b90a2dd4b68e4342b3a38b0d10ccff9a (127.0.61.1:43745), d1bfd566b5df4d4db14378793b6a2f65 (127.0.61.2:42715) I0916 07:55:37.029891 603 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "2d457d87556c49d1a61e3a4f63de087d" candidate_uuid: "e4c9a84955dd4a63aea4a185fd53f118" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b90a2dd4b68e4342b3a38b0d10ccff9a" I0916 07:55:37.030295 603 raft_consensus.cc:2890] T 2d457d87556c49d1a61e3a4f63de087d P b90a2dd4b68e4342b3a38b0d10ccff9a [term 0 FOLLOWER]: Advancing to term 1 I0916 07:55:37.029999 671 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "2d457d87556c49d1a61e3a4f63de087d" candidate_uuid: "e4c9a84955dd4a63aea4a185fd53f118" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1bfd566b5df4d4db14378793b6a2f65" I0916 07:55:37.030660 671 raft_consensus.cc:2890] T 2d457d87556c49d1a61e3a4f63de087d P d1bfd566b5df4d4db14378793b6a2f65 [term 0 FOLLOWER]: Advancing to term 1 I0916 07:55:37.036443 671 raft_consensus.cc:2350] T 2d457d87556c49d1a61e3a4f63de087d P d1bfd566b5df4d4db14378793b6a2f65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4c9a84955dd4a63aea4a185fd53f118 in term 1. I0916 07:55:37.036447 603 raft_consensus.cc:2350] T 2d457d87556c49d1a61e3a4f63de087d P b90a2dd4b68e4342b3a38b0d10ccff9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4c9a84955dd4a63aea4a185fd53f118 in term 1. I0916 07:55:37.037381 701 leader_election.cc:310] T 2d457d87556c49d1a61e3a4f63de087d P e4c9a84955dd4a63aea4a185fd53f118 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1bfd566b5df4d4db14378793b6a2f65, e4c9a84955dd4a63aea4a185fd53f118; no voters: I0916 07:55:37.038014 783 raft_consensus.cc:2644] T 2d457d87556c49d1a61e3a4f63de087d P e4c9a84955dd4a63aea4a185fd53f118 [term 1 FOLLOWER]: Leader election won for term 1 I0916 07:55:37.039244 783 raft_consensus.cc:667] T 2d457d87556c49d1a61e3a4f63de087d P e4c9a84955dd4a63aea4a185fd53f118 [term 1 LEADER]: Becoming Leader. State: Replica: e4c9a84955dd4a63aea4a185fd53f118, State: Running, Role: LEADER I0916 07:55:37.040074 783 consensus_queue.cc:206] T 2d457d87556c49d1a61e3a4f63de087d P e4c9a84955dd4a63aea4a185fd53f118 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c9a84955dd4a63aea4a185fd53f118" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 46265 } } peers { permanent_uuid: "b90a2dd4b68e4342b3a38b0d10ccff9a" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 43745 } } peers { permanent_uuid: "d1bfd566b5df4d4db14378793b6a2f65" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 42715 } } I0916 07:55:37.047094 519 catalog_manager.cc:4120] T 2d457d87556c49d1a61e3a4f63de087d P e4c9a84955dd4a63aea4a185fd53f118 reported cstate change: term changed from 0 to 1, leader changed from to e4c9a84955dd4a63aea4a185fd53f118 (127.0.61.3). New cstate: current_term: 1 leader_uuid: "e4c9a84955dd4a63aea4a185fd53f118" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c9a84955dd4a63aea4a185fd53f118" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 46265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b90a2dd4b68e4342b3a38b0d10ccff9a" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 43745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1bfd566b5df4d4db14378793b6a2f65" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 42715 } health_report { overall_health: UNKNOWN } } } I0916 07:55:37.447427 783 consensus_queue.cc:984] T 2d457d87556c49d1a61e3a4f63de087d P e4c9a84955dd4a63aea4a185fd53f118 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1bfd566b5df4d4db14378793b6a2f65" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 42715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0916 07:55:37.459456 783 consensus_queue.cc:984] T 2d457d87556c49d1a61e3a4f63de087d P e4c9a84955dd4a63aea4a185fd53f118 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b90a2dd4b68e4342b3a38b0d10ccff9a" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 43745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0916 07:55:38.104166 244 tablet_replica.cc:273] T 2d457d87556c49d1a61e3a4f63de087d P b90a2dd4b68e4342b3a38b0d10ccff9a: stopping tablet replica I0916 07:55:38.104701 244 raft_consensus.cc:2147] T 2d457d87556c49d1a61e3a4f63de087d P b90a2dd4b68e4342b3a38b0d10ccff9a [term 1 FOLLOWER]: Raft consensus shutting down. I0916 07:55:38.104964 244 raft_consensus.cc:2174] T 2d457d87556c49d1a61e3a4f63de087d P b90a2dd4b68e4342b3a38b0d10ccff9a [term 1 FOLLOWER]: Raft consensus is shut down! I0916 07:55:38.567811 699 consensus_queue.cc:544] T 2d457d87556c49d1a61e3a4f63de087d P e4c9a84955dd4a63aea4a185fd53f118 [LEADER]: The tablet replica hosted on peer b90a2dd4b68e4342b3a38b0d10ccff9a has failed W0916 07:55:38.568156 699 consensus_peers.cc:458] T 2d457d87556c49d1a61e3a4f63de087d P e4c9a84955dd4a63aea4a185fd53f118 -> Peer b90a2dd4b68e4342b3a38b0d10ccff9a (127.0.61.1:43745): Couldn't send request to peer b90a2dd4b68e4342b3a38b0d10ccff9a. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry. I0916 07:55:38.631418 244 tablet_server.cc:159] TabletServer@127.0.61.1:43745 shutting down... I0916 07:55:38.648802 244 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0916 07:55:38.664541 244 tablet_server.cc:173] TabletServer@127.0.61.1:43745 shutdown complete. I0916 07:55:38.670065 244 tablet_server.cc:159] TabletServer@127.0.61.2:42715 shutting down... I0916 07:55:38.683964 244 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0916 07:55:38.684280 244 tablet_replica.cc:273] T 2d457d87556c49d1a61e3a4f63de087d P d1bfd566b5df4d4db14378793b6a2f65: stopping tablet replica I0916 07:55:38.684630 244 raft_consensus.cc:2147] T 2d457d87556c49d1a61e3a4f63de087d P d1bfd566b5df4d4db14378793b6a2f65 [term 1 FOLLOWER]: Raft consensus shutting down. I0916 07:55:38.684875 244 raft_consensus.cc:2174] T 2d457d87556c49d1a61e3a4f63de087d P d1bfd566b5df4d4db14378793b6a2f65 [term 1 FOLLOWER]: Raft consensus is shut down! I0916 07:55:38.702446 244 tablet_server.cc:173] TabletServer@127.0.61.2:42715 shutdown complete. I0916 07:55:38.708039 244 tablet_server.cc:159] TabletServer@127.0.61.3:46265 shutting down... I0916 07:55:38.721820 244 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0916 07:55:38.722098 244 tablet_replica.cc:273] T 2d457d87556c49d1a61e3a4f63de087d P e4c9a84955dd4a63aea4a185fd53f118: stopping tablet replica I0916 07:55:38.722389 244 raft_consensus.cc:2147] T 2d457d87556c49d1a61e3a4f63de087d P e4c9a84955dd4a63aea4a185fd53f118 [term 1 LEADER]: Raft consensus shutting down. I0916 07:55:38.722924 244 raft_consensus.cc:2174] T 2d457d87556c49d1a61e3a4f63de087d P e4c9a84955dd4a63aea4a185fd53f118 [term 1 FOLLOWER]: Raft consensus is shut down! I0916 07:55:38.740197 244 tablet_server.cc:173] TabletServer@127.0.61.3:46265 shutdown complete. I0916 07:55:38.746182 244 master.cc:247] Master@127.0.61.62:39659 shutting down... I0916 07:55:38.758280 244 raft_consensus.cc:2147] T 00000000000000000000000000000000 P fd268f40e6b840f3a8b2334c84e40210 [term 1 LEADER]: Raft consensus shutting down. I0916 07:55:38.758601 244 raft_consensus.cc:2174] T 00000000000000000000000000000000 P fd268f40e6b840f3a8b2334c84e40210 [term 1 FOLLOWER]: Raft consensus is shut down! I0916 07:55:38.758780 244 tablet_replica.cc:273] T 00000000000000000000000000000000 P fd268f40e6b840f3a8b2334c84e40210: stopping tablet replica I0916 07:55:38.774732 244 master.cc:258] Master@127.0.61.62:39659 shutdown complete. [ OK ] TsTabletManagerITest.ReportOnReplicaHealthStatus (2636 ms) [ RUN ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters I0916 07:55:38.791988 244 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.0.61.62:35269 I0916 07:55:38.802570 244 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 45016us I0916 07:55:38.802912 244 fs_manager.cc:263] Metadata directory not provided I0916 07:55:38.803050 244 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0916 07:55:38.803256 244 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0916 07:55:38.803355 244 server_base.cc:453] Attempting to create new FS layout instead I0916 07:55:38.804301 244 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568620531643032-244/minicluster-data/master-0-root/instance: uuid: "01ad7865e3974566b2bf81b1ad5185a4" format_stamp: "Formatted at 2019-09-16 07:55:38 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:38.811877 244 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s I0916 07:55:38.813174 244 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0916 07:55:38.819344 244 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0916 07:55:38.819504 244 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568620531643032-244/minicluster-data/master-0-root uuid: "01ad7865e3974566b2bf81b1ad5185a4" format_stamp: "Formatted at 2019-09-16 07:55:38 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:38.819744 244 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568620531643032-244/minicluster-data/master-0-root metadata directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568620531643032-244/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568620531643032-244/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0916 07:55:38.864658 244 rpc_server.cc:205] RPC server started. Bound to: 127.0.61.62:35269 I0916 07:55:38.866312 244 webserver.cc:374] Webserver started at http://127.0.61.62:45789/ using document root and password file I0916 07:55:38.867813 860 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0916 07:55:38.874868 860 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 01ad7865e3974566b2bf81b1ad5185a4: Bootstrap starting. I0916 07:55:38.877063 860 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 01ad7865e3974566b2bf81b1ad5185a4: No blocks or log segments found. Creating new log. I0916 07:55:38.879660 860 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 01ad7865e3974566b2bf81b1ad5185a4: No bootstrap required, opened a new log I0916 07:55:38.881043 860 raft_consensus.cc:340] T 00000000000000000000000000000000 P 01ad7865e3974566b2bf81b1ad5185a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01ad7865e3974566b2bf81b1ad5185a4" member_type: VOTER } I0916 07:55:38.881371 860 raft_consensus.cc:366] T 00000000000000000000000000000000 P 01ad7865e3974566b2bf81b1ad5185a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0916 07:55:38.881515 860 raft_consensus.cc:705] T 00000000000000000000000000000000 P 01ad7865e3974566b2bf81b1ad5185a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01ad7865e3974566b2bf81b1ad5185a4, State: Initialized, Role: FOLLOWER I0916 07:55:38.881888 860 consensus_queue.cc:229] T 00000000000000000000000000000000 P 01ad7865e3974566b2bf81b1ad5185a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01ad7865e3974566b2bf81b1ad5185a4" member_type: VOTER } I0916 07:55:38.882244 860 raft_consensus.cc:380] T 00000000000000000000000000000000 P 01ad7865e3974566b2bf81b1ad5185a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0916 07:55:38.882398 860 raft_consensus.cc:472] T 00000000000000000000000000000000 P 01ad7865e3974566b2bf81b1ad5185a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0916 07:55:38.882586 860 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 01ad7865e3974566b2bf81b1ad5185a4 [term 0 FOLLOWER]: Advancing to term 1 I0916 07:55:38.885830 860 raft_consensus.cc:494] T 00000000000000000000000000000000 P 01ad7865e3974566b2bf81b1ad5185a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01ad7865e3974566b2bf81b1ad5185a4" member_type: VOTER } I0916 07:55:38.886219 860 leader_election.cc:310] T 00000000000000000000000000000000 P 01ad7865e3974566b2bf81b1ad5185a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01ad7865e3974566b2bf81b1ad5185a4; no voters: I0916 07:55:38.887570 862 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 01ad7865e3974566b2bf81b1ad5185a4 [term 1 FOLLOWER]: Leader election won for term 1 I0916 07:55:38.888224 860 leader_election.cc:296] T 00000000000000000000000000000000 P 01ad7865e3974566b2bf81b1ad5185a4 [CANDIDATE]: Term 1 election: Requested vote from peers I0916 07:55:38.888756 862 raft_consensus.cc:667] T 00000000000000000000000000000000 P 01ad7865e3974566b2bf81b1ad5185a4 [term 1 LEADER]: Becoming Leader. State: Replica: 01ad7865e3974566b2bf81b1ad5185a4, State: Running, Role: LEADER I0916 07:55:38.889291 862 consensus_queue.cc:206] T 00000000000000000000000000000000 P 01ad7865e3974566b2bf81b1ad5185a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01ad7865e3974566b2bf81b1ad5185a4" member_type: VOTER } I0916 07:55:38.892714 863 sys_catalog.cc:340] T 00000000000000000000000000000000 P 01ad7865e3974566b2bf81b1ad5185a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01ad7865e3974566b2bf81b1ad5185a4. Latest consensus state: current_term: 1 leader_uuid: "01ad7865e3974566b2bf81b1ad5185a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01ad7865e3974566b2bf81b1ad5185a4" member_type: VOTER } } I0916 07:55:38.893297 863 sys_catalog.cc:343] T 00000000000000000000000000000000 P 01ad7865e3974566b2bf81b1ad5185a4 [sys.catalog]: This master's current role is: LEADER I0916 07:55:38.894306 864 sys_catalog.cc:340] T 00000000000000000000000000000000 P 01ad7865e3974566b2bf81b1ad5185a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01ad7865e3974566b2bf81b1ad5185a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01ad7865e3974566b2bf81b1ad5185a4" member_type: VOTER } } I0916 07:55:38.894862 864 sys_catalog.cc:343] T 00000000000000000000000000000000 P 01ad7865e3974566b2bf81b1ad5185a4 [sys.catalog]: This master's current role is: LEADER I0916 07:55:38.895539 860 sys_catalog.cc:340] T 00000000000000000000000000000000 P 01ad7865e3974566b2bf81b1ad5185a4 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "01ad7865e3974566b2bf81b1ad5185a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01ad7865e3974566b2bf81b1ad5185a4" member_type: VOTER } } I0916 07:55:38.896059 860 sys_catalog.cc:343] T 00000000000000000000000000000000 P 01ad7865e3974566b2bf81b1ad5185a4 [sys.catalog]: This master's current role is: LEADER I0916 07:55:38.897662 860 sys_catalog.cc:428] T 00000000000000000000000000000000 P 01ad7865e3974566b2bf81b1ad5185a4 [sys.catalog]: configured and running, proceeding with master startup. I0916 07:55:38.898264 868 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0916 07:55:38.900948 869 catalog_manager.cc:569] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting I0916 07:55:38.901162 868 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0916 07:55:38.904685 244 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0916 07:55:38.920751 868 catalog_manager.cc:981] Generated new certificate authority record I0916 07:55:38.922233 868 catalog_manager.cc:1106] Loading token signing keys... I0916 07:55:38.943692 868 catalog_manager.cc:4435] T 00000000000000000000000000000000 P 01ad7865e3974566b2bf81b1ad5185a4: Generated new TSK 0 I0916 07:55:38.981268 244 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 45016us I0916 07:55:38.981652 244 fs_manager.cc:263] Metadata directory not provided I0916 07:55:38.981801 244 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0916 07:55:38.981992 244 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0916 07:55:38.982100 244 server_base.cc:453] Attempting to create new FS layout instead I0916 07:55:38.983036 244 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568620531643032-244/minicluster-data/ts-0-root/instance: uuid: "745f880e7aac4edfa96aca2c35eceebe" format_stamp: "Formatted at 2019-09-16 07:55:38 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:38.993198 244 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.011s sys 0.001s I0916 07:55:38.994684 244 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0916 07:55:39.000887 244 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0916 07:55:39.001073 244 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568620531643032-244/minicluster-data/ts-0-root uuid: "745f880e7aac4edfa96aca2c35eceebe" format_stamp: "Formatted at 2019-09-16 07:55:38 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:39.001317 244 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568620531643032-244/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568620531643032-244/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568620531643032-244/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0916 07:55:39.030388 244 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0916 07:55:39.030552 244 ts_tablet_manager.cc:403] Registered 0 tablets I0916 07:55:39.064955 244 rpc_server.cc:205] RPC server started. Bound to: 127.0.61.1:39751 I0916 07:55:39.066650 244 webserver.cc:374] Webserver started at http://127.0.61.1:44605/ using document root and password file I0916 07:55:39.078619 935 heartbeater.cc:322] Connected to a master server at 127.0.61.62:35269 I0916 07:55:39.078902 935 heartbeater.cc:406] Registering TS with master... I0916 07:55:39.079294 935 heartbeater.cc:455] Master 127.0.61.62:35269 requested a full tablet report, sending... I0916 07:55:39.080788 827 ts_manager.cc:161] Registered new tserver with Master: 745f880e7aac4edfa96aca2c35eceebe (127.0.61.1:39751) I0916 07:55:39.081763 244 internal_mini_cluster.cc:328] 1 TS(s) registered with all masters after 0.012577009s I0916 07:55:39.082352 827 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45168 I0916 07:55:39.099915 244 tablet_server.cc:159] TabletServer@127.0.61.1:39751 shutting down... I0916 07:55:39.125849 244 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0916 07:55:39.141054 244 tablet_server.cc:173] TabletServer@127.0.61.1:39751 shutdown complete. I0916 07:55:39.156913 244 tablet_server.cc:90] deduplicated master addresses: 127.0.61.62:35269 I0916 07:55:39.157466 244 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 45516us I0916 07:55:39.157783 244 fs_manager.cc:263] Metadata directory not provided I0916 07:55:39.157903 244 fs_manager.cc:266] Using existing metadata directory in first data directory I0916 07:55:39.160348 244 fs_manager.cc:399] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s I0916 07:55:39.161307 244 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0916 07:55:39.167699 244 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0916 07:55:39.167887 244 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568620531643032-244/minicluster-data/ts-0-root uuid: "745f880e7aac4edfa96aca2c35eceebe" format_stamp: "Formatted at 2019-09-16 07:55:38 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:39.168164 244 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568620531643032-244/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568620531643032-244/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568620531643032-244/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0916 07:55:39.204160 244 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0916 07:55:39.204330 244 ts_tablet_manager.cc:403] Registered 0 tablets I0916 07:55:39.243146 244 rpc_server.cc:205] RPC server started. Bound to: 127.0.61.1:39751 I0916 07:55:39.245117 244 webserver.cc:374] Webserver started at http://127.0.61.1:44605/ using document root and password file I0916 07:55:39.249572 244 tablet_server.cc:159] TabletServer@127.0.61.1:39751 shutting down... I0916 07:55:39.270246 1009 heartbeater.cc:322] Connected to a master server at 127.0.61.62:35269 I0916 07:55:39.270685 1009 heartbeater.cc:406] Registering TS with master... I0916 07:55:39.271158 1009 heartbeater.cc:455] Master 127.0.61.62:35269 requested a full tablet report, sending... I0916 07:55:39.272940 827 ts_manager.cc:161] Re-registered known tserver with Master: 745f880e7aac4edfa96aca2c35eceebe (127.0.61.1:39751) I0916 07:55:39.274159 827 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45172 I0916 07:55:39.276250 244 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0916 07:55:39.291296 244 tablet_server.cc:173] TabletServer@127.0.61.1:39751 shutdown complete. I0916 07:55:39.297099 244 master.cc:247] Master@127.0.61.62:35269 shutting down... I0916 07:55:39.308357 244 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 01ad7865e3974566b2bf81b1ad5185a4 [term 1 LEADER]: Raft consensus shutting down. I0916 07:55:39.308739 244 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 01ad7865e3974566b2bf81b1ad5185a4 [term 1 FOLLOWER]: Raft consensus is shut down! I0916 07:55:39.309047 244 tablet_replica.cc:273] T 00000000000000000000000000000000 P 01ad7865e3974566b2bf81b1ad5185a4: stopping tablet replica I0916 07:55:39.326951 244 master.cc:258] Master@127.0.61.62:35269 shutdown complete. [ OK ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters (550 ms) [ RUN ] TsTabletManagerITest.TestTableStats I0916 07:55:39.342872 244 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.0.61.62:33603,127.0.61.61:37305,127.0.61.60:37785 I0916 07:55:39.353441 244 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 45516us I0916 07:55:39.353775 244 fs_manager.cc:263] Metadata directory not provided I0916 07:55:39.353894 244 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0916 07:55:39.354064 244 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0916 07:55:39.354158 244 server_base.cc:453] Attempting to create new FS layout instead I0916 07:55:39.355015 244 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/master-0-root/instance: uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" format_stamp: "Formatted at 2019-09-16 07:55:39 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:39.362546 244 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s I0916 07:55:39.363776 244 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0916 07:55:39.369704 244 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0916 07:55:39.369858 244 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/master-0-root uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" format_stamp: "Formatted at 2019-09-16 07:55:39 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:39.370070 244 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/master-0-root metadata directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0916 07:55:39.429659 244 rpc_server.cc:205] RPC server started. Bound to: 127.0.61.62:33603 I0916 07:55:39.431599 244 webserver.cc:374] Webserver started at http://127.0.61.62:36877/ using document root and password file I0916 07:55:39.433751 1070 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0916 07:55:39.439867 1070 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } has no permanent_uuid. Determining permanent_uuid... I0916 07:55:39.448779 244 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 45516us I0916 07:55:39.449122 244 fs_manager.cc:263] Metadata directory not provided I0916 07:55:39.449265 244 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0916 07:55:39.449450 244 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0916 07:55:39.449548 244 server_base.cc:453] Attempting to create new FS layout instead I0916 07:55:39.450692 244 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/master-1-root/instance: uuid: "5fe1f662b7a54c0f823ca2d656131416" format_stamp: "Formatted at 2019-09-16 07:55:39 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:39.453423 1070 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } has no permanent_uuid. Determining permanent_uuid... W0916 07:55:39.455094 1070 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.0.61.61:37305: Network error: Client connection negotiation failed: client connection to 127.0.61.61:37305: connect: Connection refused (error 111) I0916 07:55:39.459599 244 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.010s sys 0.001s I0916 07:55:39.461006 244 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0916 07:55:39.467234 244 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0916 07:55:39.467394 244 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/master-1-root uuid: "5fe1f662b7a54c0f823ca2d656131416" format_stamp: "Formatted at 2019-09-16 07:55:39 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:39.467674 244 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/master-1-root metadata directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/master-1-root 1 data directories: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/master-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0916 07:55:39.507495 1070 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } attempt: 1 W0916 07:55:39.509521 1070 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.0.61.61:37305: Network error: Client connection negotiation failed: client connection to 127.0.61.61:37305: connect: Connection refused (error 111) I0916 07:55:39.528828 244 rpc_server.cc:205] RPC server started. Bound to: 127.0.61.61:37305 I0916 07:55:39.530465 244 webserver.cc:374] Webserver started at http://127.0.61.61:38341/ using document root and password file I0916 07:55:39.532131 1127 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0916 07:55:39.536325 1127 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } has no permanent_uuid. Determining permanent_uuid... I0916 07:55:39.545466 244 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 45516us I0916 07:55:39.545819 244 fs_manager.cc:263] Metadata directory not provided I0916 07:55:39.545966 244 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0916 07:55:39.546161 244 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0916 07:55:39.546304 244 server_base.cc:453] Attempting to create new FS layout instead I0916 07:55:39.547438 244 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/master-2-root/instance: uuid: "d933e518c06d4c7d8b4f743167c2c643" format_stamp: "Formatted at 2019-09-16 07:55:39 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:39.549813 1127 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } has no permanent_uuid. Determining permanent_uuid... I0916 07:55:39.559937 244 fs_manager.cc:503] Time spent creating directory manager: real 0.012s user 0.010s sys 0.004s I0916 07:55:39.561693 244 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0916 07:55:39.562027 1070 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } attempt: 2 I0916 07:55:39.564415 1127 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } has no permanent_uuid. Determining permanent_uuid... W0916 07:55:39.566133 1127 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.0.61.60:37785: Network error: Client connection negotiation failed: client connection to 127.0.61.60:37785: connect: Connection refused (error 111) I0916 07:55:39.570979 244 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0916 07:55:39.571290 244 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/master-2-root uuid: "d933e518c06d4c7d8b4f743167c2c643" format_stamp: "Formatted at 2019-09-16 07:55:39 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:39.571622 244 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/master-2-root metadata directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/master-2-root 1 data directories: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/master-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0916 07:55:39.572511 1070 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } has no permanent_uuid. Determining permanent_uuid... W0916 07:55:39.574303 1070 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.0.61.60:37785: Network error: Client connection negotiation failed: client connection to 127.0.61.60:37785: connect: Connection refused (error 111) I0916 07:55:39.586568 1127 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } attempt: 1 W0916 07:55:39.588501 1127 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.0.61.60:37785: Network error: Client connection negotiation failed: client connection to 127.0.61.60:37785: connect: Connection refused (error 111) I0916 07:55:39.624656 244 rpc_server.cc:205] RPC server started. Bound to: 127.0.61.60:37785 I0916 07:55:39.626250 244 webserver.cc:374] Webserver started at http://127.0.61.60:45039/ using document root and password file I0916 07:55:39.627146 244 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0916 07:55:39.627913 1184 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0916 07:55:39.630913 1127 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } attempt: 2 I0916 07:55:39.632175 1184 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } has no permanent_uuid. Determining permanent_uuid... I0916 07:55:39.635761 1070 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } attempt: 1 I0916 07:55:39.648283 1184 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } has no permanent_uuid. Determining permanent_uuid... I0916 07:55:39.651023 1127 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416: Bootstrap starting. I0916 07:55:39.653789 1127 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416: No blocks or log segments found. Creating new log. I0916 07:55:39.658118 1127 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416: No bootstrap required, opened a new log I0916 07:55:39.658938 1184 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } has no permanent_uuid. Determining permanent_uuid... I0916 07:55:39.660048 1127 raft_consensus.cc:340] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } I0916 07:55:39.661058 1127 raft_consensus.cc:366] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0916 07:55:39.661331 1127 raft_consensus.cc:705] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fe1f662b7a54c0f823ca2d656131416, State: Initialized, Role: FOLLOWER I0916 07:55:39.661883 1127 consensus_queue.cc:229] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } I0916 07:55:39.664063 1070 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f: Bootstrap starting. I0916 07:55:39.663812 1127 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:39.664618 1127 sys_catalog.cc:343] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: This master's current role is: FOLLOWER I0916 07:55:39.664021 1189 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:39.665010 1189 sys_catalog.cc:343] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: This master's current role is: FOLLOWER I0916 07:55:39.665650 1127 sys_catalog.cc:428] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: configured and running, proceeding with master startup. I0916 07:55:39.667587 1070 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f: No blocks or log segments found. Creating new log. I0916 07:55:39.671154 1070 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f: No bootstrap required, opened a new log W0916 07:55:39.671290 1190 catalog_manager.cc:1144] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0916 07:55:39.671555 1190 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0916 07:55:39.672600 1070 raft_consensus.cc:340] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } I0916 07:55:39.673100 1070 raft_consensus.cc:366] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0916 07:55:39.673272 1070 raft_consensus.cc:705] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd1e33dc408e40c7a8bae2850d5a6b6f, State: Initialized, Role: FOLLOWER I0916 07:55:39.674156 1184 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643: Bootstrap starting. I0916 07:55:39.673868 1070 consensus_queue.cc:229] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } I0916 07:55:39.675431 1070 sys_catalog.cc:340] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:39.675917 1070 sys_catalog.cc:343] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [sys.catalog]: This master's current role is: FOLLOWER I0916 07:55:39.675658 1192 sys_catalog.cc:340] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:39.676416 1192 sys_catalog.cc:343] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [sys.catalog]: This master's current role is: FOLLOWER I0916 07:55:39.676721 1070 sys_catalog.cc:428] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [sys.catalog]: configured and running, proceeding with master startup. I0916 07:55:39.677282 1184 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643: No blocks or log segments found. Creating new log. I0916 07:55:39.677937 244 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 1 I0916 07:55:39.678164 244 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 2 W0916 07:55:39.679829 1193 catalog_manager.cc:1144] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0916 07:55:39.680039 1193 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0916 07:55:39.680294 1184 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643: No bootstrap required, opened a new log I0916 07:55:39.681668 1184 raft_consensus.cc:340] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } I0916 07:55:39.682117 1184 raft_consensus.cc:366] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0916 07:55:39.682281 1184 raft_consensus.cc:705] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d933e518c06d4c7d8b4f743167c2c643, State: Initialized, Role: FOLLOWER I0916 07:55:39.682696 1184 consensus_queue.cc:229] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } I0916 07:55:39.684005 1184 sys_catalog.cc:340] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:39.684484 1184 sys_catalog.cc:343] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [sys.catalog]: This master's current role is: FOLLOWER I0916 07:55:39.684150 1195 sys_catalog.cc:340] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:39.685019 1195 sys_catalog.cc:343] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [sys.catalog]: This master's current role is: FOLLOWER I0916 07:55:39.685253 1184 sys_catalog.cc:428] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [sys.catalog]: configured and running, proceeding with master startup. W0916 07:55:39.688146 1196 catalog_manager.cc:1144] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0916 07:55:39.688318 1196 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0916 07:55:39.697713 1189 raft_consensus.cc:472] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0916 07:55:39.698171 1189 raft_consensus.cc:494] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } I0916 07:55:39.699681 1189 leader_election.cc:296] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd1e33dc408e40c7a8bae2850d5a6b6f (127.0.61.62:33603), d933e518c06d4c7d8b4f743167c2c643 (127.0.61.60:37785) I0916 07:55:39.700575 1195 raft_consensus.cc:472] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0916 07:55:39.700695 1047 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5fe1f662b7a54c0f823ca2d656131416" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" is_pre_election: true I0916 07:55:39.701283 1047 raft_consensus.cc:2350] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fe1f662b7a54c0f823ca2d656131416 in term 0. I0916 07:55:39.700992 1195 raft_consensus.cc:494] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } I0916 07:55:39.701968 1161 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5fe1f662b7a54c0f823ca2d656131416" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d933e518c06d4c7d8b4f743167c2c643" is_pre_election: true I0916 07:55:39.702239 1078 leader_election.cc:310] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fe1f662b7a54c0f823ca2d656131416, fd1e33dc408e40c7a8bae2850d5a6b6f; no voters: I0916 07:55:39.702550 1161 raft_consensus.cc:2350] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fe1f662b7a54c0f823ca2d656131416 in term 0. I0916 07:55:39.703042 1189 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0916 07:55:39.703114 1047 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d933e518c06d4c7d8b4f743167c2c643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" is_pre_election: true I0916 07:55:39.703335 1189 raft_consensus.cc:472] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0916 07:55:39.703284 1104 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d933e518c06d4c7d8b4f743167c2c643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fe1f662b7a54c0f823ca2d656131416" is_pre_election: true I0916 07:55:39.703656 1047 raft_consensus.cc:2350] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d933e518c06d4c7d8b4f743167c2c643 in term 0. I0916 07:55:39.703714 1189 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 0 FOLLOWER]: Advancing to term 1 I0916 07:55:39.704018 1195 leader_election.cc:296] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd1e33dc408e40c7a8bae2850d5a6b6f (127.0.61.62:33603), 5fe1f662b7a54c0f823ca2d656131416 (127.0.61.61:37305) I0916 07:55:39.704572 1135 leader_election.cc:310] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d933e518c06d4c7d8b4f743167c2c643, fd1e33dc408e40c7a8bae2850d5a6b6f; no voters: I0916 07:55:39.704952 244 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 45516us I0916 07:55:39.705320 244 fs_manager.cc:263] Metadata directory not provided I0916 07:55:39.705309 1195 raft_consensus.cc:2644] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0916 07:55:39.705631 244 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0916 07:55:39.705677 1195 raft_consensus.cc:472] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0916 07:55:39.705904 244 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0916 07:55:39.705974 1195 raft_consensus.cc:2890] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 0 FOLLOWER]: Advancing to term 1 I0916 07:55:39.706104 244 server_base.cc:453] Attempting to create new FS layout instead I0916 07:55:39.707250 244 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-0-root/instance: uuid: "39273ba79fdb4d31abbb34278b8b1709" format_stamp: "Formatted at 2019-09-16 07:55:39 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:39.708216 1189 raft_consensus.cc:494] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } I0916 07:55:39.709120 1104 raft_consensus.cc:2280] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d933e518c06d4c7d8b4f743167c2c643 in current term 1: Already voted for candidate 5fe1f662b7a54c0f823ca2d656131416 in this term. I0916 07:55:39.710310 1047 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5fe1f662b7a54c0f823ca2d656131416" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" I0916 07:55:39.710767 1047 raft_consensus.cc:2890] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 0 FOLLOWER]: Advancing to term 1 I0916 07:55:39.711067 1195 raft_consensus.cc:494] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } I0916 07:55:39.712527 1195 leader_election.cc:296] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [CANDIDATE]: Term 1 election: Requested vote from peers fd1e33dc408e40c7a8bae2850d5a6b6f (127.0.61.62:33603), 5fe1f662b7a54c0f823ca2d656131416 (127.0.61.61:37305) I0916 07:55:39.713361 1104 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d933e518c06d4c7d8b4f743167c2c643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fe1f662b7a54c0f823ca2d656131416" I0916 07:55:39.713393 1046 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d933e518c06d4c7d8b4f743167c2c643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" I0916 07:55:39.714187 1104 raft_consensus.cc:2280] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d933e518c06d4c7d8b4f743167c2c643 in current term 1: Already voted for candidate 5fe1f662b7a54c0f823ca2d656131416 in this term. I0916 07:55:39.714910 1161 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5fe1f662b7a54c0f823ca2d656131416" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d933e518c06d4c7d8b4f743167c2c643" I0916 07:55:39.715430 1161 raft_consensus.cc:2280] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5fe1f662b7a54c0f823ca2d656131416 in current term 1: Already voted for candidate d933e518c06d4c7d8b4f743167c2c643 in this term. I0916 07:55:39.715788 1047 raft_consensus.cc:2350] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fe1f662b7a54c0f823ca2d656131416 in term 1. I0916 07:55:39.715862 1189 leader_election.cc:296] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [CANDIDATE]: Term 1 election: Requested vote from peers fd1e33dc408e40c7a8bae2850d5a6b6f (127.0.61.62:33603), d933e518c06d4c7d8b4f743167c2c643 (127.0.61.60:37785) I0916 07:55:39.716498 1046 raft_consensus.cc:2325] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d933e518c06d4c7d8b4f743167c2c643 for term 1 because replica is already servicing an update from a current leader or another vote. I0916 07:55:39.717763 1135 leader_election.cc:310] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d933e518c06d4c7d8b4f743167c2c643; no voters: 5fe1f662b7a54c0f823ca2d656131416, fd1e33dc408e40c7a8bae2850d5a6b6f I0916 07:55:39.717907 1078 leader_election.cc:310] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5fe1f662b7a54c0f823ca2d656131416, fd1e33dc408e40c7a8bae2850d5a6b6f; no voters: d933e518c06d4c7d8b4f743167c2c643 I0916 07:55:39.718474 1195 raft_consensus.cc:2592] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority I0916 07:55:39.719060 1189 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 1 FOLLOWER]: Leader election won for term 1 I0916 07:55:39.720212 1189 raft_consensus.cc:667] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 1 LEADER]: Becoming Leader. State: Replica: 5fe1f662b7a54c0f823ca2d656131416, State: Running, Role: LEADER I0916 07:55:39.720779 1189 consensus_queue.cc:206] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } I0916 07:55:39.722996 244 fs_manager.cc:503] Time spent creating directory manager: real 0.015s user 0.013s sys 0.000s I0916 07:55:39.725687 1197 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fe1f662b7a54c0f823ca2d656131416. Latest consensus state: current_term: 1 leader_uuid: "5fe1f662b7a54c0f823ca2d656131416" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:39.726166 244 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0916 07:55:39.726342 1197 sys_catalog.cc:343] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: This master's current role is: LEADER I0916 07:55:39.729243 1200 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0916 07:55:39.731544 1200 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0916 07:55:39.749651 244 fs_manager.cc:419] Time spent opening block manager: real 0.013s user 0.000s sys 0.001s I0916 07:55:39.749904 244 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-0-root uuid: "39273ba79fdb4d31abbb34278b8b1709" format_stamp: "Formatted at 2019-09-16 07:55:39 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:39.750211 244 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0916 07:55:39.775847 1161 raft_consensus.cc:1184] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 1 FOLLOWER]: Refusing update from remote peer 5fe1f662b7a54c0f823ca2d656131416: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0916 07:55:39.776368 1047 raft_consensus.cc:1184] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 1 FOLLOWER]: Refusing update from remote peer 5fe1f662b7a54c0f823ca2d656131416: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0916 07:55:39.777174 1197 consensus_queue.cc:984] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0916 07:55:39.777734 1189 consensus_queue.cc:984] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0916 07:55:39.789816 1192 sys_catalog.cc:340] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fe1f662b7a54c0f823ca2d656131416. Latest consensus state: current_term: 1 leader_uuid: "5fe1f662b7a54c0f823ca2d656131416" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:39.790802 1192 sys_catalog.cc:343] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [sys.catalog]: This master's current role is: FOLLOWER I0916 07:55:39.791390 1195 sys_catalog.cc:340] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fe1f662b7a54c0f823ca2d656131416. Latest consensus state: current_term: 1 leader_uuid: "5fe1f662b7a54c0f823ca2d656131416" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:39.791965 1195 sys_catalog.cc:343] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [sys.catalog]: This master's current role is: FOLLOWER I0916 07:55:39.799748 1206 mvcc.cc:203] Tried to move safe_time back from 6425069730910695424 to 6425069730702548992. Current Snapshot: MvccSnapshot[committed={T|T < 6425069730910695424}] I0916 07:55:39.804137 1197 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5fe1f662b7a54c0f823ca2d656131416" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:39.805402 1197 sys_catalog.cc:343] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: This master's current role is: LEADER I0916 07:55:39.806216 1189 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5fe1f662b7a54c0f823ca2d656131416" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:39.807219 1189 sys_catalog.cc:343] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: This master's current role is: LEADER I0916 07:55:39.809674 1200 catalog_manager.cc:981] Generated new certificate authority record I0916 07:55:39.809293 1195 sys_catalog.cc:340] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5fe1f662b7a54c0f823ca2d656131416" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:39.810072 1195 sys_catalog.cc:343] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [sys.catalog]: This master's current role is: FOLLOWER I0916 07:55:39.811614 1200 catalog_manager.cc:1106] Loading token signing keys... I0916 07:55:39.811296 1192 sys_catalog.cc:340] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5fe1f662b7a54c0f823ca2d656131416" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:39.812007 1192 sys_catalog.cc:343] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [sys.catalog]: This master's current role is: FOLLOWER I0916 07:55:39.814395 244 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0916 07:55:39.814586 244 ts_tablet_manager.cc:403] Registered 0 tablets I0916 07:55:39.835608 1200 catalog_manager.cc:4435] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416: Generated new TSK 0 I0916 07:55:39.871186 244 rpc_server.cc:205] RPC server started. Bound to: 127.0.61.1:40619 I0916 07:55:39.873172 244 webserver.cc:374] Webserver started at http://127.0.61.1:34949/ using document root and password file I0916 07:55:39.901855 1273 heartbeater.cc:322] Connected to a master server at 127.0.61.60:37785 I0916 07:55:39.902310 1273 heartbeater.cc:406] Registering TS with master... I0916 07:55:39.902825 1273 heartbeater.cc:455] Master 127.0.61.60:37785 requested a full tablet report, sending... I0916 07:55:39.905550 1274 heartbeater.cc:322] Connected to a master server at 127.0.61.61:37305 I0916 07:55:39.905839 1274 heartbeater.cc:406] Registering TS with master... I0916 07:55:39.906412 1274 heartbeater.cc:455] Master 127.0.61.61:37305 requested a full tablet report, sending... I0916 07:55:39.907385 1275 heartbeater.cc:322] Connected to a master server at 127.0.61.62:33603 I0916 07:55:39.907728 1275 heartbeater.cc:406] Registering TS with master... I0916 07:55:39.908025 1151 ts_manager.cc:161] Registered new tserver with Master: 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619) I0916 07:55:39.908372 1275 heartbeater.cc:455] Master 127.0.61.62:33603 requested a full tablet report, sending... I0916 07:55:39.909051 1094 ts_manager.cc:161] Registered new tserver with Master: 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619) I0916 07:55:39.912312 1094 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42742 I0916 07:55:39.912360 1037 ts_manager.cc:161] Registered new tserver with Master: 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619) I0916 07:55:39.914337 244 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 45516us I0916 07:55:39.914748 244 fs_manager.cc:263] Metadata directory not provided I0916 07:55:39.914957 244 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0916 07:55:39.915159 244 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0916 07:55:39.915258 244 server_base.cc:453] Attempting to create new FS layout instead I0916 07:55:39.916091 244 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-1-root/instance: uuid: "bce8dbcd72a840bb8256dee0715b175f" format_stamp: "Formatted at 2019-09-16 07:55:39 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:39.925771 244 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.012s sys 0.000s I0916 07:55:39.927141 244 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0916 07:55:39.936151 244 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s I0916 07:55:39.936350 244 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-1-root uuid: "bce8dbcd72a840bb8256dee0715b175f" format_stamp: "Formatted at 2019-09-16 07:55:39 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:39.936643 244 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0916 07:55:39.966260 244 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0916 07:55:39.966454 244 ts_tablet_manager.cc:403] Registered 0 tablets I0916 07:55:40.015403 1274 heartbeater.cc:447] Master 127.0.61.61:37305 was elected leader, sending a full tablet report... I0916 07:55:40.069452 244 rpc_server.cc:205] RPC server started. Bound to: 127.0.61.2:39401 I0916 07:55:40.072805 244 webserver.cc:374] Webserver started at http://127.0.61.2:37889/ using document root and password file I0916 07:55:40.103806 1344 heartbeater.cc:322] Connected to a master server at 127.0.61.60:37785 I0916 07:55:40.104171 1344 heartbeater.cc:406] Registering TS with master... I0916 07:55:40.105108 1344 heartbeater.cc:455] Master 127.0.61.60:37785 requested a full tablet report, sending... I0916 07:55:40.107547 1151 ts_manager.cc:161] Registered new tserver with Master: bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401) I0916 07:55:40.108654 1345 heartbeater.cc:322] Connected to a master server at 127.0.61.61:37305 I0916 07:55:40.108937 1345 heartbeater.cc:406] Registering TS with master... I0916 07:55:40.109469 1345 heartbeater.cc:455] Master 127.0.61.61:37305 requested a full tablet report, sending... I0916 07:55:40.111768 1347 heartbeater.cc:322] Connected to a master server at 127.0.61.62:33603 I0916 07:55:40.112066 1347 heartbeater.cc:406] Registering TS with master... I0916 07:55:40.112521 1347 heartbeater.cc:455] Master 127.0.61.62:33603 requested a full tablet report, sending... I0916 07:55:40.112561 1094 ts_manager.cc:161] Registered new tserver with Master: bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401) I0916 07:55:40.114264 1037 ts_manager.cc:161] Registered new tserver with Master: bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401) I0916 07:55:40.114475 1094 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42748 I0916 07:55:40.116791 244 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 46016us I0916 07:55:40.118099 244 fs_manager.cc:263] Metadata directory not provided I0916 07:55:40.118351 244 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0916 07:55:40.118640 244 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0916 07:55:40.118830 244 server_base.cc:453] Attempting to create new FS layout instead I0916 07:55:40.119944 244 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-2-root/instance: uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" format_stamp: "Formatted at 2019-09-16 07:55:40 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:40.130946 244 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.010s sys 0.000s I0916 07:55:40.132560 244 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0916 07:55:40.141283 244 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0916 07:55:40.141465 244 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-2-root uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" format_stamp: "Formatted at 2019-09-16 07:55:40 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:40.141698 244 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0916 07:55:40.181838 244 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0916 07:55:40.182013 244 ts_tablet_manager.cc:403] Registered 0 tablets I0916 07:55:40.219082 1345 heartbeater.cc:447] Master 127.0.61.61:37305 was elected leader, sending a full tablet report... I0916 07:55:40.278820 244 rpc_server.cc:205] RPC server started. Bound to: 127.0.61.3:45451 I0916 07:55:40.281805 244 webserver.cc:374] Webserver started at http://127.0.61.3:39227/ using document root and password file I0916 07:55:40.317220 1415 heartbeater.cc:322] Connected to a master server at 127.0.61.60:37785 I0916 07:55:40.317544 1415 heartbeater.cc:406] Registering TS with master... I0916 07:55:40.318128 1415 heartbeater.cc:455] Master 127.0.61.60:37785 requested a full tablet report, sending... I0916 07:55:40.320293 1416 heartbeater.cc:322] Connected to a master server at 127.0.61.61:37305 I0916 07:55:40.320571 1151 ts_manager.cc:161] Registered new tserver with Master: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:40.320708 1416 heartbeater.cc:406] Registering TS with master... I0916 07:55:40.322190 1416 heartbeater.cc:455] Master 127.0.61.61:37305 requested a full tablet report, sending... I0916 07:55:40.323874 1094 ts_manager.cc:161] Registered new tserver with Master: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:40.324453 1418 heartbeater.cc:322] Connected to a master server at 127.0.61.62:33603 I0916 07:55:40.324841 1418 heartbeater.cc:406] Registering TS with master... I0916 07:55:40.325371 1418 heartbeater.cc:455] Master 127.0.61.62:33603 requested a full tablet report, sending... I0916 07:55:40.328888 244 internal_mini_cluster.cc:328] 3 TS(s) registered with all masters after 0.022469012s I0916 07:55:40.328310 1037 ts_manager.cc:161] Registered new tserver with Master: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:40.328291 1094 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42754 I0916 07:55:40.359596 1094 catalog_manager.cc:1440] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42762: name: "test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } W0916 07:55:40.360651 1094 catalog_manager.cc:1605] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0916 07:55:40.434126 1416 heartbeater.cc:447] Master 127.0.61.61:37305 was elected leader, sending a full tablet report... I0916 07:55:40.470368 1240 tablet_service.cc:1022] Processing CreateTablet for tablet 6b5196040b094455ad661e9ab9e379c7 (table=test-table [id=074536fca6144499906b7c13e035c370]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0916 07:55:40.471190 1240 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 6b5196040b094455ad661e9ab9e379c7. 1 dirs total, 0 dirs full, 0 dirs failed I0916 07:55:40.479053 1382 tablet_service.cc:1022] Processing CreateTablet for tablet 6b5196040b094455ad661e9ab9e379c7 (table=test-table [id=074536fca6144499906b7c13e035c370]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0916 07:55:40.479849 1238 tablet_service.cc:1022] Processing CreateTablet for tablet e5cce06208f44e4fa384db94c0b624ae (table=test-table [id=074536fca6144499906b7c13e035c370]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0916 07:55:40.481256 1382 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 6b5196040b094455ad661e9ab9e379c7. 1 dirs total, 0 dirs full, 0 dirs failed I0916 07:55:40.481101 1380 tablet_service.cc:1022] Processing CreateTablet for tablet e5cce06208f44e4fa384db94c0b624ae (table=test-table [id=074536fca6144499906b7c13e035c370]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0916 07:55:40.479055 1239 tablet_service.cc:1022] Processing CreateTablet for tablet e02eeaf40c1d4a299b70f4a8f1514e31 (table=test-table [id=074536fca6144499906b7c13e035c370]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0916 07:55:40.482879 1380 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet e5cce06208f44e4fa384db94c0b624ae. 1 dirs total, 0 dirs full, 0 dirs failed I0916 07:55:40.485102 1239 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet e02eeaf40c1d4a299b70f4a8f1514e31. 1 dirs total, 0 dirs full, 0 dirs failed I0916 07:55:40.487007 1309 tablet_service.cc:1022] Processing CreateTablet for tablet e5cce06208f44e4fa384db94c0b624ae (table=test-table [id=074536fca6144499906b7c13e035c370]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0916 07:55:40.487936 1310 tablet_service.cc:1022] Processing CreateTablet for tablet e02eeaf40c1d4a299b70f4a8f1514e31 (table=test-table [id=074536fca6144499906b7c13e035c370]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0916 07:55:40.486980 1381 tablet_service.cc:1022] Processing CreateTablet for tablet e02eeaf40c1d4a299b70f4a8f1514e31 (table=test-table [id=074536fca6144499906b7c13e035c370]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0916 07:55:40.489814 1381 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet e02eeaf40c1d4a299b70f4a8f1514e31. 1 dirs total, 0 dirs full, 0 dirs failed I0916 07:55:40.486479 1311 tablet_service.cc:1022] Processing CreateTablet for tablet 6b5196040b094455ad661e9ab9e379c7 (table=test-table [id=074536fca6144499906b7c13e035c370]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0916 07:55:40.498085 1311 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 6b5196040b094455ad661e9ab9e379c7. 1 dirs total, 0 dirs full, 0 dirs failed I0916 07:55:40.492959 1238 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet e5cce06208f44e4fa384db94c0b624ae. 1 dirs total, 0 dirs full, 0 dirs failed I0916 07:55:40.499058 1309 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet e5cce06208f44e4fa384db94c0b624ae. 1 dirs total, 0 dirs full, 0 dirs failed I0916 07:55:40.504570 1310 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet e02eeaf40c1d4a299b70f4a8f1514e31. 1 dirs total, 0 dirs full, 0 dirs failed I0916 07:55:40.532407 1444 tablet_bootstrap.cc:439] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709: Bootstrap starting. I0916 07:55:40.565251 1446 tablet_bootstrap.cc:439] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Bootstrap starting. I0916 07:55:40.565634 1444 tablet_bootstrap.cc:590] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709: No blocks or log segments found. Creating new log. I0916 07:55:40.565317 1445 tablet_bootstrap.cc:439] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f: Bootstrap starting. I0916 07:55:40.572232 1446 tablet_bootstrap.cc:590] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: No blocks or log segments found. Creating new log. I0916 07:55:40.574021 1445 tablet_bootstrap.cc:590] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f: No blocks or log segments found. Creating new log. I0916 07:55:40.576969 1444 tablet_bootstrap.cc:439] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709: No bootstrap required, opened a new log I0916 07:55:40.577692 1444 ts_tablet_manager.cc:1085] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709: Time spent bootstrapping tablet: real 0.046s user 0.010s sys 0.007s I0916 07:55:40.581140 1446 tablet_bootstrap.cc:439] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: No bootstrap required, opened a new log I0916 07:55:40.581260 1445 tablet_bootstrap.cc:439] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f: No bootstrap required, opened a new log I0916 07:55:40.581667 1446 ts_tablet_manager.cc:1085] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.004s I0916 07:55:40.581316 1444 raft_consensus.cc:340] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } I0916 07:55:40.581728 1445 ts_tablet_manager.cc:1085] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.000s I0916 07:55:40.582108 1444 raft_consensus.cc:366] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0916 07:55:40.582345 1444 raft_consensus.cc:705] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39273ba79fdb4d31abbb34278b8b1709, State: Initialized, Role: FOLLOWER I0916 07:55:40.583604 1444 consensus_queue.cc:229] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } I0916 07:55:40.584686 1446 raft_consensus.cc:340] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:40.584852 1445 raft_consensus.cc:340] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } I0916 07:55:40.585566 1445 raft_consensus.cc:366] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0916 07:55:40.585883 1445 raft_consensus.cc:705] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bce8dbcd72a840bb8256dee0715b175f, State: Initialized, Role: FOLLOWER I0916 07:55:40.585475 1446 raft_consensus.cc:366] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0916 07:55:40.587011 1446 raft_consensus.cc:705] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, State: Initialized, Role: FOLLOWER I0916 07:55:40.587687 1446 consensus_queue.cc:229] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:40.589023 1444 ts_tablet_manager.cc:1113] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709: Time spent starting tablet: real 0.011s user 0.006s sys 0.003s I0916 07:55:40.586452 1445 consensus_queue.cc:229] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } I0916 07:55:40.589937 1444 tablet_bootstrap.cc:439] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709: Bootstrap starting. I0916 07:55:40.602986 1444 tablet_bootstrap.cc:590] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709: No blocks or log segments found. Creating new log. I0916 07:55:40.605175 1445 ts_tablet_manager.cc:1113] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f: Time spent starting tablet: real 0.023s user 0.001s sys 0.009s I0916 07:55:40.607132 1445 tablet_bootstrap.cc:439] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f: Bootstrap starting. I0916 07:55:40.613586 1445 tablet_bootstrap.cc:590] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f: No blocks or log segments found. Creating new log. I0916 07:55:40.621613 1446 ts_tablet_manager.cc:1113] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Time spent starting tablet: real 0.040s user 0.005s sys 0.005s I0916 07:55:40.622512 1446 tablet_bootstrap.cc:439] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Bootstrap starting. I0916 07:55:40.624871 1445 tablet_bootstrap.cc:439] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f: No bootstrap required, opened a new log I0916 07:55:40.625444 1445 ts_tablet_manager.cc:1085] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.009s I0916 07:55:40.627487 1445 raft_consensus.cc:340] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:40.628173 1445 raft_consensus.cc:366] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0916 07:55:40.628475 1445 raft_consensus.cc:705] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bce8dbcd72a840bb8256dee0715b175f, State: Initialized, Role: FOLLOWER I0916 07:55:40.629168 1445 consensus_queue.cc:229] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:40.631909 1446 tablet_bootstrap.cc:590] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: No blocks or log segments found. Creating new log. I0916 07:55:40.632917 1444 tablet_bootstrap.cc:439] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709: No bootstrap required, opened a new log I0916 07:55:40.633276 1445 ts_tablet_manager.cc:1113] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s I0916 07:55:40.633688 1444 ts_tablet_manager.cc:1085] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709: Time spent bootstrapping tablet: real 0.044s user 0.006s sys 0.004s I0916 07:55:40.634035 1445 tablet_bootstrap.cc:439] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f: Bootstrap starting. I0916 07:55:40.637274 1444 raft_consensus.cc:340] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:40.638144 1444 raft_consensus.cc:366] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0916 07:55:40.638424 1444 raft_consensus.cc:705] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39273ba79fdb4d31abbb34278b8b1709, State: Initialized, Role: FOLLOWER I0916 07:55:40.639276 1445 tablet_bootstrap.cc:590] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f: No blocks or log segments found. Creating new log. I0916 07:55:40.639140 1444 consensus_queue.cc:229] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:40.645227 1450 raft_consensus.cc:472] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0916 07:55:40.645694 1450 raft_consensus.cc:494] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } I0916 07:55:40.647830 1450 leader_election.cc:296] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451), bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401) I0916 07:55:40.655475 1445 tablet_bootstrap.cc:439] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f: No bootstrap required, opened a new log I0916 07:55:40.656123 1445 ts_tablet_manager.cc:1085] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.011s I0916 07:55:40.658169 1444 ts_tablet_manager.cc:1113] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709: Time spent starting tablet: real 0.024s user 0.002s sys 0.004s I0916 07:55:40.659786 1444 tablet_bootstrap.cc:439] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709: Bootstrap starting. I0916 07:55:40.666160 1445 raft_consensus.cc:340] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:40.666869 1445 raft_consensus.cc:366] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0916 07:55:40.667054 1445 raft_consensus.cc:705] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bce8dbcd72a840bb8256dee0715b175f, State: Initialized, Role: FOLLOWER I0916 07:55:40.667297 1446 tablet_bootstrap.cc:439] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: No bootstrap required, opened a new log I0916 07:55:40.667847 1446 ts_tablet_manager.cc:1085] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Time spent bootstrapping tablet: real 0.045s user 0.012s sys 0.009s I0916 07:55:40.667515 1445 consensus_queue.cc:229] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:40.670895 1452 raft_consensus.cc:472] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0916 07:55:40.671499 1445 ts_tablet_manager.cc:1113] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s I0916 07:55:40.671408 1452 raft_consensus.cc:494] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:40.681797 1444 tablet_bootstrap.cc:590] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709: No blocks or log segments found. Creating new log. I0916 07:55:40.682206 1446 raft_consensus.cc:340] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } I0916 07:55:40.683071 1446 raft_consensus.cc:366] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0916 07:55:40.683327 1446 raft_consensus.cc:705] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, State: Initialized, Role: FOLLOWER I0916 07:55:40.681939 1451 raft_consensus.cc:472] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0916 07:55:40.683902 1451 raft_consensus.cc:494] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:40.685880 1446 consensus_queue.cc:229] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } I0916 07:55:40.686884 1452 leader_election.cc:296] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619), bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401) I0916 07:55:40.703815 1450 raft_consensus.cc:472] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0916 07:55:40.704269 1450 raft_consensus.cc:494] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:40.706421 1450 leader_election.cc:296] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401), 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:40.715207 1193 catalog_manager.cc:1142] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f: acquiring CA information for follower catalog manager: success I0916 07:55:40.716574 1451 leader_election.cc:296] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619), 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:40.752197 1451 raft_consensus.cc:472] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0916 07:55:40.765480 1452 raft_consensus.cc:472] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0916 07:55:40.765967 1452 raft_consensus.cc:494] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } I0916 07:55:40.794996 1452 leader_election.cc:296] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619), bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401) I0916 07:55:40.800179 1196 catalog_manager.cc:1142] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643: acquiring CA information for follower catalog manager: success I0916 07:55:40.761612 1451 raft_consensus.cc:494] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } I0916 07:55:40.804884 1451 leader_election.cc:296] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619), 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:40.807853 1193 catalog_manager.cc:1170] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0916 07:55:40.791296 1446 ts_tablet_manager.cc:1113] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Time spent starting tablet: real 0.123s user 0.004s sys 0.004s I0916 07:55:40.812687 1392 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6b5196040b094455ad661e9ab9e379c7" candidate_uuid: "39273ba79fdb4d31abbb34278b8b1709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" is_pre_election: true I0916 07:55:40.818143 1392 raft_consensus.cc:2350] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39273ba79fdb4d31abbb34278b8b1709 in term 0. I0916 07:55:40.819942 1391 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "39273ba79fdb4d31abbb34278b8b1709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" is_pre_election: true I0916 07:55:40.822789 1205 leader_election.cc:310] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, 39273ba79fdb4d31abbb34278b8b1709; no voters: I0916 07:55:40.823730 1450 raft_consensus.cc:2644] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0916 07:55:40.824034 1450 raft_consensus.cc:472] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0916 07:55:40.824395 1450 raft_consensus.cc:2890] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 0 FOLLOWER]: Advancing to term 1 W0916 07:55:40.824208 1205 leader_election.cc:348] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451): Illegal state: must be running to vote when last-logged opid is not known I0916 07:55:40.828442 1446 tablet_bootstrap.cc:439] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Bootstrap starting. I0916 07:55:40.835180 1450 raft_consensus.cc:494] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } I0916 07:55:40.839378 1196 catalog_manager.cc:1170] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0916 07:55:40.841807 1391 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6b5196040b094455ad661e9ab9e379c7" candidate_uuid: "39273ba79fdb4d31abbb34278b8b1709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" I0916 07:55:40.845394 1391 raft_consensus.cc:2890] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 0 FOLLOWER]: Advancing to term 1 I0916 07:55:40.844563 1321 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6b5196040b094455ad661e9ab9e379c7" candidate_uuid: "39273ba79fdb4d31abbb34278b8b1709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce8dbcd72a840bb8256dee0715b175f" is_pre_election: true I0916 07:55:40.842541 1459 raft_consensus.cc:472] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0916 07:55:40.854774 1250 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e5cce06208f44e4fa384db94c0b624ae" candidate_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39273ba79fdb4d31abbb34278b8b1709" is_pre_election: true I0916 07:55:40.851783 1450 leader_election.cc:296] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 1 election: Requested vote from peers 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451), bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401) I0916 07:55:40.871224 1459 raft_consensus.cc:494] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } W0916 07:55:40.873402 1354 leader_election.cc:348] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619): Illegal state: must be running to vote when last-logged opid is not known I0916 07:55:40.874852 1459 leader_election.cc:296] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619), 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:40.874946 1249 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6b5196040b094455ad661e9ab9e379c7" candidate_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39273ba79fdb4d31abbb34278b8b1709" is_pre_election: true I0916 07:55:40.878367 1249 raft_consensus.cc:2280] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 in current term 1: Already voted for candidate 39273ba79fdb4d31abbb34278b8b1709 in this term. I0916 07:55:40.853756 1321 raft_consensus.cc:2350] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39273ba79fdb4d31abbb34278b8b1709 in term 0. W0916 07:55:40.925706 1353 leader_election.cc:341] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401): Timed out: connection negotiation to 127.0.61.2:39401 for RPC RequestConsensusVote timed out after 0.235s (ON_OUTBOUND_QUEUE) I0916 07:55:40.926239 1353 leader_election.cc:310] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90; no voters: 39273ba79fdb4d31abbb34278b8b1709, bce8dbcd72a840bb8256dee0715b175f I0916 07:55:40.927130 1452 raft_consensus.cc:2592] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0916 07:55:40.947218 1391 raft_consensus.cc:2350] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39273ba79fdb4d31abbb34278b8b1709 in term 1. W0916 07:55:40.955597 1202 leader_election.cc:341] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401): Timed out: RequestConsensusVote RPC to 127.0.61.2:39401 timed out after 0.245s (SENT) I0916 07:55:40.963237 1459 raft_consensus.cc:472] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0916 07:55:40.964116 1320 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6b5196040b094455ad661e9ab9e379c7" candidate_uuid: "39273ba79fdb4d31abbb34278b8b1709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce8dbcd72a840bb8256dee0715b175f" I0916 07:55:40.956601 1321 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "39273ba79fdb4d31abbb34278b8b1709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce8dbcd72a840bb8256dee0715b175f" is_pre_election: true I0916 07:55:40.972048 1459 raft_consensus.cc:494] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:40.974653 1459 leader_election.cc:296] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619), 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:40.975164 1320 raft_consensus.cc:2890] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 0 FOLLOWER]: Advancing to term 1 I0916 07:55:40.975415 1446 tablet_bootstrap.cc:590] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: No blocks or log segments found. Creating new log. I0916 07:55:40.964721 1205 leader_election.cc:310] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, 39273ba79fdb4d31abbb34278b8b1709; no voters: I0916 07:55:40.976581 1321 raft_consensus.cc:2350] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39273ba79fdb4d31abbb34278b8b1709 in term 0. I0916 07:55:40.977246 1450 raft_consensus.cc:2644] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 1 FOLLOWER]: Leader election won for term 1 W0916 07:55:40.979146 1202 leader_election.cc:341] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401): Timed out: RequestConsensusVote RPC to 127.0.61.2:39401 timed out after 0.264s (SENT) I0916 07:55:40.979449 1202 leader_election.cc:310] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39273ba79fdb4d31abbb34278b8b1709; no voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, bce8dbcd72a840bb8256dee0715b175f I0916 07:55:40.985661 1444 tablet_bootstrap.cc:439] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709: No bootstrap required, opened a new log I0916 07:55:40.986433 1444 ts_tablet_manager.cc:1085] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709: Time spent bootstrapping tablet: real 0.327s user 0.006s sys 0.017s I0916 07:55:40.989723 1392 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6b5196040b094455ad661e9ab9e379c7" candidate_uuid: "bce8dbcd72a840bb8256dee0715b175f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" is_pre_election: true I0916 07:55:40.990053 1444 raft_consensus.cc:340] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:40.990887 1392 raft_consensus.cc:2280] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bce8dbcd72a840bb8256dee0715b175f in current term 1: Already voted for candidate 39273ba79fdb4d31abbb34278b8b1709 in this term. W0916 07:55:40.992566 1283 leader_election.cc:341] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619): Timed out: RequestConsensusVote RPC to 127.0.61.1:40619 timed out after 0.278s (SENT) W0916 07:55:40.993531 1285 leader_election.cc:341] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451): Timed out: RequestConsensusVote RPC to 127.0.61.3:45451 timed out after 0.278s (SENT) I0916 07:55:40.992251 1444 raft_consensus.cc:366] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0916 07:55:40.994503 1285 leader_election.cc:310] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bce8dbcd72a840bb8256dee0715b175f; no voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, 39273ba79fdb4d31abbb34278b8b1709 I0916 07:55:40.995450 1444 raft_consensus.cc:705] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39273ba79fdb4d31abbb34278b8b1709, State: Initialized, Role: FOLLOWER I0916 07:55:40.995512 1392 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e5cce06208f44e4fa384db94c0b624ae" candidate_uuid: "bce8dbcd72a840bb8256dee0715b175f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" is_pre_election: true I0916 07:55:40.996183 1321 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6b5196040b094455ad661e9ab9e379c7" candidate_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce8dbcd72a840bb8256dee0715b175f" is_pre_election: true I0916 07:55:40.996400 1459 raft_consensus.cc:2592] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0916 07:55:40.996109 1444 consensus_queue.cc:229] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:40.988777 1391 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "bce8dbcd72a840bb8256dee0715b175f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" is_pre_election: true I0916 07:55:40.991941 1320 raft_consensus.cc:2350] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39273ba79fdb4d31abbb34278b8b1709 in term 1. I0916 07:55:40.996243 1392 raft_consensus.cc:2350] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bce8dbcd72a840bb8256dee0715b175f in term 0. I0916 07:55:40.998159 1321 raft_consensus.cc:2325] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 for term 1 because replica is already servicing an update from a current leader or another vote. I0916 07:55:40.999588 1392 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "bce8dbcd72a840bb8256dee0715b175f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" is_pre_election: true I0916 07:55:41.001052 1285 leader_election.cc:310] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, bce8dbcd72a840bb8256dee0715b175f; no voters: W0916 07:55:41.002035 1285 leader_election.cc:348] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451): Illegal state: must be running to vote when last-logged opid is not known I0916 07:55:41.002163 1459 raft_consensus.cc:2644] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 0 FOLLOWER]: Leader pre-election won for term 1 I0916 07:55:41.003300 1459 raft_consensus.cc:472] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0916 07:55:41.003459 1465 raft_consensus.cc:472] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5fe1f662b7a54c0f823ca2d656131416) I0916 07:55:41.003669 1459 raft_consensus.cc:2890] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 0 FOLLOWER]: Advancing to term 1 W0916 07:55:41.004272 1354 leader_election.cc:341] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619): Timed out: RequestConsensusVote RPC to 127.0.61.1:40619 timed out after 0.210s (SENT) I0916 07:55:41.005232 1354 leader_election.cc:310] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90; no voters: 39273ba79fdb4d31abbb34278b8b1709, bce8dbcd72a840bb8256dee0715b175f I0916 07:55:41.006054 1452 raft_consensus.cc:2592] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0916 07:55:41.011162 1459 raft_consensus.cc:494] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:41.015020 1392 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e5cce06208f44e4fa384db94c0b624ae" candidate_uuid: "bce8dbcd72a840bb8256dee0715b175f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" I0916 07:55:41.015558 1392 raft_consensus.cc:2890] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 0 FOLLOWER]: Advancing to term 1 I0916 07:55:41.004274 1465 raft_consensus.cc:494] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } I0916 07:55:41.019652 1392 raft_consensus.cc:2350] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bce8dbcd72a840bb8256dee0715b175f in term 1. I0916 07:55:41.020622 1470 raft_consensus.cc:2592] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0916 07:55:41.020704 1285 leader_election.cc:310] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, bce8dbcd72a840bb8256dee0715b175f; no voters: I0916 07:55:41.022068 1470 raft_consensus.cc:472] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0916 07:55:41.022547 1451 raft_consensus.cc:2644] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 1 FOLLOWER]: Leader election won for term 1 I0916 07:55:41.022449 1470 raft_consensus.cc:494] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:41.023300 1465 leader_election.cc:296] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5fe1f662b7a54c0f823ca2d656131416 (127.0.61.61:37305), d933e518c06d4c7d8b4f743167c2c643 (127.0.61.60:37785) I0916 07:55:41.021700 1459 leader_election.cc:296] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 1 election: Requested vote from peers 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619), 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:41.024848 1161 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "d933e518c06d4c7d8b4f743167c2c643" is_pre_election: true I0916 07:55:41.025758 1161 raft_consensus.cc:2311] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fd1e33dc408e40c7a8bae2850d5a6b6f for term 2 because replica is either leader or believes a valid leader to be alive. I0916 07:55:41.025982 1104 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "5fe1f662b7a54c0f823ca2d656131416" is_pre_election: true I0916 07:55:41.026470 1470 leader_election.cc:296] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401), 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:41.026669 1104 raft_consensus.cc:2311] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate fd1e33dc408e40c7a8bae2850d5a6b6f for term 2 because replica is either leader or believes a valid leader to be alive. I0916 07:55:41.027940 1020 leader_election.cc:310] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd1e33dc408e40c7a8bae2850d5a6b6f; no voters: 5fe1f662b7a54c0f823ca2d656131416, d933e518c06d4c7d8b4f743167c2c643 I0916 07:55:41.028331 1451 raft_consensus.cc:667] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 1 LEADER]: Becoming Leader. State: Replica: bce8dbcd72a840bb8256dee0715b175f, State: Running, Role: LEADER I0916 07:55:41.029045 1465 raft_consensus.cc:2592] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0916 07:55:41.028980 1451 consensus_queue.cc:206] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:41.021502 1444 ts_tablet_manager.cc:1113] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709: Time spent starting tablet: real 0.035s user 0.007s sys 0.003s I0916 07:55:41.030921 1450 raft_consensus.cc:667] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 1 LEADER]: Becoming Leader. State: Replica: 39273ba79fdb4d31abbb34278b8b1709, State: Running, Role: LEADER I0916 07:55:41.032784 1392 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "39273ba79fdb4d31abbb34278b8b1709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" is_pre_election: true W0916 07:55:41.033538 1202 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.61.2:39401, user_credentials={real_user=slave}} blocked reactor thread for 54384.9us I0916 07:55:41.035610 1249 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "bce8dbcd72a840bb8256dee0715b175f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39273ba79fdb4d31abbb34278b8b1709" is_pre_election: true I0916 07:55:41.036239 1249 raft_consensus.cc:2350] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bce8dbcd72a840bb8256dee0715b175f in term 0. I0916 07:55:41.038116 1321 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "39273ba79fdb4d31abbb34278b8b1709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce8dbcd72a840bb8256dee0715b175f" is_pre_election: true I0916 07:55:41.039674 1092 catalog_manager.cc:4120] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f reported cstate change: term changed from 0 to 1, leader changed from to bce8dbcd72a840bb8256dee0715b175f (127.0.61.2). New cstate: current_term: 1 leader_uuid: "bce8dbcd72a840bb8256dee0715b175f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } health_report { overall_health: UNKNOWN } } } I0916 07:55:41.040884 1249 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6b5196040b094455ad661e9ab9e379c7" candidate_uuid: "bce8dbcd72a840bb8256dee0715b175f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39273ba79fdb4d31abbb34278b8b1709" is_pre_election: true I0916 07:55:41.038761 1321 raft_consensus.cc:2350] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39273ba79fdb4d31abbb34278b8b1709 in term 0. W0916 07:55:41.043298 1205 leader_election.cc:348] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451): Illegal state: must be running to vote when last-logged opid is not known I0916 07:55:41.039670 1450 consensus_queue.cc:206] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } I0916 07:55:41.046705 1202 leader_election.cc:310] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39273ba79fdb4d31abbb34278b8b1709, bce8dbcd72a840bb8256dee0715b175f; no voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 I0916 07:55:41.047210 1446 tablet_bootstrap.cc:439] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: No bootstrap required, opened a new log I0916 07:55:41.049437 1249 raft_consensus.cc:2311] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate bce8dbcd72a840bb8256dee0715b175f for term 1 because replica is either leader or believes a valid leader to be alive. I0916 07:55:41.049417 1469 raft_consensus.cc:2644] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0916 07:55:41.049989 1469 raft_consensus.cc:472] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0916 07:55:41.050371 1469 raft_consensus.cc:2890] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 0 FOLLOWER]: Advancing to term 1 I0916 07:55:41.051904 1249 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e5cce06208f44e4fa384db94c0b624ae" candidate_uuid: "bce8dbcd72a840bb8256dee0715b175f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39273ba79fdb4d31abbb34278b8b1709" is_pre_election: true W0916 07:55:41.052809 1283 leader_election.cc:341] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619): Timed out: RequestConsensusVote RPC to 127.0.61.1:40619 timed out after 0.208s (SENT) I0916 07:55:41.053154 1283 leader_election.cc:310] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bce8dbcd72a840bb8256dee0715b175f; no voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, 39273ba79fdb4d31abbb34278b8b1709 I0916 07:55:41.054127 1249 raft_consensus.cc:2350] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bce8dbcd72a840bb8256dee0715b175f in term 0. I0916 07:55:41.057649 1451 raft_consensus.cc:2592] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0916 07:55:41.048681 1446 ts_tablet_manager.cc:1085] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Time spent bootstrapping tablet: real 0.229s user 0.039s sys 0.008s I0916 07:55:41.066756 1249 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "bce8dbcd72a840bb8256dee0715b175f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39273ba79fdb4d31abbb34278b8b1709" is_pre_election: true I0916 07:55:41.067018 1250 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e5cce06208f44e4fa384db94c0b624ae" candidate_uuid: "bce8dbcd72a840bb8256dee0715b175f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39273ba79fdb4d31abbb34278b8b1709" I0916 07:55:41.067615 1250 raft_consensus.cc:2890] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 0 FOLLOWER]: Advancing to term 1 I0916 07:55:41.073561 1446 raft_consensus.cc:340] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:41.074615 1446 raft_consensus.cc:366] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0916 07:55:41.076213 1446 raft_consensus.cc:705] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, State: Initialized, Role: FOLLOWER I0916 07:55:41.081573 1250 raft_consensus.cc:2350] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bce8dbcd72a840bb8256dee0715b175f in term 1. I0916 07:55:41.081955 1446 consensus_queue.cc:229] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:41.087461 1452 raft_consensus.cc:472] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0916 07:55:41.088024 1452 raft_consensus.cc:494] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:41.090469 1452 leader_election.cc:296] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401), 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619) I0916 07:55:41.092733 1321 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce8dbcd72a840bb8256dee0715b175f" is_pre_election: true I0916 07:55:41.093504 1321 raft_consensus.cc:2350] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 in term 0. I0916 07:55:41.096711 1353 leader_election.cc:310] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, bce8dbcd72a840bb8256dee0715b175f; no voters: I0916 07:55:41.097407 1452 raft_consensus.cc:2644] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0916 07:55:41.097792 1452 raft_consensus.cc:472] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0916 07:55:41.097659 1250 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39273ba79fdb4d31abbb34278b8b1709" is_pre_election: true I0916 07:55:41.098166 1452 raft_consensus.cc:2890] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 0 FOLLOWER]: Advancing to term 1 I0916 07:55:41.119165 1469 raft_consensus.cc:494] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:41.120630 1249 raft_consensus.cc:2280] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bce8dbcd72a840bb8256dee0715b175f in current term 1: Already voted for candidate 39273ba79fdb4d31abbb34278b8b1709 in this term. I0916 07:55:41.121956 1250 raft_consensus.cc:2325] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 for term 1 because replica is already servicing an update from a current leader or another vote. I0916 07:55:41.122498 1283 leader_election.cc:310] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bce8dbcd72a840bb8256dee0715b175f; no voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, 39273ba79fdb4d31abbb34278b8b1709 I0916 07:55:41.123545 1451 raft_consensus.cc:2890] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 0 FOLLOWER]: Advancing to term 1 I0916 07:55:41.153255 1450 consensus_queue.cc:984] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0916 07:55:41.154742 1451 raft_consensus.cc:2592] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0916 07:55:41.156040 1392 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "39273ba79fdb4d31abbb34278b8b1709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" I0916 07:55:41.162912 1469 leader_election.cc:296] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 1 election: Requested vote from peers bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401), 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:41.173322 1321 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "39273ba79fdb4d31abbb34278b8b1709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce8dbcd72a840bb8256dee0715b175f" I0916 07:55:41.172392 1446 ts_tablet_manager.cc:1113] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Time spent starting tablet: real 0.113s user 0.008s sys 0.015s I0916 07:55:41.180984 1093 catalog_manager.cc:4120] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 reported cstate change: term changed from 0 to 1, leader changed from to 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1). New cstate: current_term: 1 leader_uuid: "39273ba79fdb4d31abbb34278b8b1709" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } health_report { overall_health: UNKNOWN } } } I0916 07:55:41.171880 1451 consensus_queue.cc:984] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s I0916 07:55:41.164572 1452 raft_consensus.cc:494] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:41.186260 1450 consensus_queue.cc:984] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s I0916 07:55:41.195128 1392 raft_consensus.cc:2280] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 39273ba79fdb4d31abbb34278b8b1709 in current term 1: Already voted for candidate 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 in this term. I0916 07:55:41.194819 1321 raft_consensus.cc:2350] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39273ba79fdb4d31abbb34278b8b1709 in term 1. I0916 07:55:41.206955 1202 leader_election.cc:310] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39273ba79fdb4d31abbb34278b8b1709, bce8dbcd72a840bb8256dee0715b175f; no voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 I0916 07:55:41.211326 1452 leader_election.cc:296] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 1 election: Requested vote from peers bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401), 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619) I0916 07:55:41.211663 1321 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce8dbcd72a840bb8256dee0715b175f" I0916 07:55:41.214206 1321 raft_consensus.cc:2280] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 in current term 1: Already voted for candidate 39273ba79fdb4d31abbb34278b8b1709 in this term. I0916 07:55:41.211477 1450 raft_consensus.cc:2644] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 1 FOLLOWER]: Leader election won for term 1 I0916 07:55:41.215937 1450 raft_consensus.cc:667] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 1 LEADER]: Becoming Leader. State: Replica: 39273ba79fdb4d31abbb34278b8b1709, State: Running, Role: LEADER I0916 07:55:41.216801 1450 consensus_queue.cc:206] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:41.229530 1250 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39273ba79fdb4d31abbb34278b8b1709" I0916 07:55:41.233610 1451 consensus_queue.cc:984] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [LEADER]: Connected to new peer: Peer: permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0916 07:55:41.239914 1250 raft_consensus.cc:2311] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 1 LEADER]: Leader election vote request: Denying vote to candidate 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 for term 1 because replica is either leader or believes a valid leader to be alive. I0916 07:55:41.257926 1093 catalog_manager.cc:4120] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 reported cstate change: term changed from 0 to 1, leader changed from to 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1). New cstate: current_term: 1 leader_uuid: "39273ba79fdb4d31abbb34278b8b1709" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } health_report { overall_health: UNKNOWN } } } I0916 07:55:41.277511 1354 leader_election.cc:310] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90; no voters: 39273ba79fdb4d31abbb34278b8b1709, bce8dbcd72a840bb8256dee0715b175f I0916 07:55:41.294973 1452 raft_consensus.cc:2592] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority I0916 07:55:41.327102 1469 consensus_queue.cc:984] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0916 07:55:41.337945 1469 consensus_queue.cc:984] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0916 07:55:41.450093 1452 raft_consensus.cc:472] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bce8dbcd72a840bb8256dee0715b175f) I0916 07:55:41.450668 1452 raft_consensus.cc:494] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:41.455001 1250 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e5cce06208f44e4fa384db94c0b624ae" candidate_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39273ba79fdb4d31abbb34278b8b1709" is_pre_election: true I0916 07:55:41.455682 1250 raft_consensus.cc:2311] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 for term 2 because replica is either leader or believes a valid leader to be alive. I0916 07:55:41.457068 1452 leader_election.cc:296] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619), bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401) I0916 07:55:41.459548 1321 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e5cce06208f44e4fa384db94c0b624ae" candidate_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bce8dbcd72a840bb8256dee0715b175f" is_pre_election: true I0916 07:55:41.461980 1321 raft_consensus.cc:2311] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 for term 2 because replica is either leader or believes a valid leader to be alive. I0916 07:55:41.463014 1353 leader_election.cc:310] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90; no voters: 39273ba79fdb4d31abbb34278b8b1709, bce8dbcd72a840bb8256dee0715b175f I0916 07:55:41.464872 1452 raft_consensus.cc:2592] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0916 07:55:41.498325 1493 raft_consensus.cc:472] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39273ba79fdb4d31abbb34278b8b1709) I0916 07:55:41.499042 1493 raft_consensus.cc:494] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } I0916 07:55:41.502352 1391 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6b5196040b094455ad661e9ab9e379c7" candidate_uuid: "bce8dbcd72a840bb8256dee0715b175f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" is_pre_election: true I0916 07:55:41.503459 1391 raft_consensus.cc:2311] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bce8dbcd72a840bb8256dee0715b175f for term 2 because replica is either leader or believes a valid leader to be alive. I0916 07:55:41.504930 1493 leader_election.cc:296] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619), 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:41.505703 1250 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6b5196040b094455ad661e9ab9e379c7" candidate_uuid: "bce8dbcd72a840bb8256dee0715b175f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39273ba79fdb4d31abbb34278b8b1709" is_pre_election: true I0916 07:55:41.506266 1250 raft_consensus.cc:2311] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate bce8dbcd72a840bb8256dee0715b175f for term 2 because replica is either leader or believes a valid leader to be alive. I0916 07:55:41.508781 1283 leader_election.cc:310] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bce8dbcd72a840bb8256dee0715b175f; no voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, 39273ba79fdb4d31abbb34278b8b1709 I0916 07:55:41.509377 1493 raft_consensus.cc:2592] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0916 07:55:41.788034 1249 tablet_service.cc:1440] Received LeaderStepDown RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" dest_uuid: "39273ba79fdb4d31abbb34278b8b1709" from {username='slave'} at 127.0.0.1:35102 I0916 07:55:41.789753 1249 raft_consensus.cc:569] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 1 LEADER]: Received request to step down I0916 07:55:41.790410 1249 raft_consensus.cc:2885] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 1 LEADER]: Stepping down as leader of term 1 I0916 07:55:41.790680 1249 raft_consensus.cc:705] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39273ba79fdb4d31abbb34278b8b1709, State: Running, Role: LEADER I0916 07:55:41.792029 1249 consensus_queue.cc:229] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:41.796301 1249 raft_consensus.cc:2890] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 1 FOLLOWER]: Advancing to term 2 I0916 07:55:41.931896 1493 raft_consensus.cc:472] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39273ba79fdb4d31abbb34278b8b1709) I0916 07:55:41.932546 1493 raft_consensus.cc:494] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:41.934132 1493 leader_election.cc:296] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619), 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:41.935209 1249 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "bce8dbcd72a840bb8256dee0715b175f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "39273ba79fdb4d31abbb34278b8b1709" is_pre_election: true I0916 07:55:41.935209 1391 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "bce8dbcd72a840bb8256dee0715b175f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" is_pre_election: true I0916 07:55:41.936177 1249 raft_consensus.cc:2350] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bce8dbcd72a840bb8256dee0715b175f in term 2. I0916 07:55:41.936336 1391 raft_consensus.cc:2311] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bce8dbcd72a840bb8256dee0715b175f for term 2 because replica is either leader or believes a valid leader to be alive. I0916 07:55:41.937476 1283 leader_election.cc:310] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39273ba79fdb4d31abbb34278b8b1709, bce8dbcd72a840bb8256dee0715b175f; no voters: I0916 07:55:41.938215 1493 raft_consensus.cc:2644] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 1 FOLLOWER]: Leader pre-election won for term 2 I0916 07:55:41.938593 1493 raft_consensus.cc:472] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 39273ba79fdb4d31abbb34278b8b1709) I0916 07:55:41.938951 1493 raft_consensus.cc:2890] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 1 FOLLOWER]: Advancing to term 2 I0916 07:55:41.943765 1493 raft_consensus.cc:494] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:41.945080 1493 leader_election.cc:296] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 2 election: Requested vote from peers 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619), 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:41.945890 1249 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "bce8dbcd72a840bb8256dee0715b175f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "39273ba79fdb4d31abbb34278b8b1709" I0916 07:55:41.945957 1391 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "bce8dbcd72a840bb8256dee0715b175f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" I0916 07:55:41.946481 1391 raft_consensus.cc:2890] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 1 FOLLOWER]: Advancing to term 2 I0916 07:55:41.950246 1249 raft_consensus.cc:2350] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bce8dbcd72a840bb8256dee0715b175f in term 2. I0916 07:55:41.950460 1391 raft_consensus.cc:2350] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bce8dbcd72a840bb8256dee0715b175f in term 2. I0916 07:55:41.951273 1283 leader_election.cc:310] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39273ba79fdb4d31abbb34278b8b1709, bce8dbcd72a840bb8256dee0715b175f; no voters: I0916 07:55:41.951920 1493 raft_consensus.cc:2644] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 2 FOLLOWER]: Leader election won for term 2 I0916 07:55:41.952261 1493 raft_consensus.cc:667] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 2 LEADER]: Becoming Leader. State: Replica: bce8dbcd72a840bb8256dee0715b175f, State: Running, Role: LEADER I0916 07:55:41.952857 1493 consensus_queue.cc:206] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:41.959553 1094 catalog_manager.cc:4120] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f reported cstate change: term changed from 1 to 2, leader changed from 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1) to bce8dbcd72a840bb8256dee0715b175f (127.0.61.2). New cstate: current_term: 2 leader_uuid: "bce8dbcd72a840bb8256dee0715b175f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } health_report { overall_health: UNKNOWN } } } I0916 07:55:42.025528 1249 tablet_service.cc:1440] Received LeaderStepDown RPC: tablet_id: "6b5196040b094455ad661e9ab9e379c7" dest_uuid: "39273ba79fdb4d31abbb34278b8b1709" from {username='slave'} at 127.0.0.1:35102 I0916 07:55:42.026041 1249 raft_consensus.cc:569] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 1 LEADER]: Received request to step down I0916 07:55:42.026337 1249 raft_consensus.cc:2885] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 1 LEADER]: Stepping down as leader of term 1 I0916 07:55:42.026530 1249 raft_consensus.cc:705] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39273ba79fdb4d31abbb34278b8b1709, State: Running, Role: LEADER I0916 07:55:42.027160 1249 consensus_queue.cc:229] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } I0916 07:55:42.027992 1249 raft_consensus.cc:2890] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 1 FOLLOWER]: Advancing to term 2 I0916 07:55:42.077906 1390 raft_consensus.cc:1184] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 2 FOLLOWER]: Refusing update from remote peer bce8dbcd72a840bb8256dee0715b175f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch) I0916 07:55:42.079187 1493 consensus_queue.cc:984] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0916 07:55:42.079663 1249 raft_consensus.cc:1184] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 2 FOLLOWER]: Refusing update from remote peer bce8dbcd72a840bb8256dee0715b175f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch) I0916 07:55:42.080832 1493 consensus_queue.cc:984] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [LEADER]: Connected to new peer: Peer: permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0916 07:55:42.194170 1518 raft_consensus.cc:472] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39273ba79fdb4d31abbb34278b8b1709) I0916 07:55:42.194665 1518 raft_consensus.cc:494] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } I0916 07:55:42.195981 1518 leader_election.cc:296] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619), bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401) I0916 07:55:42.197150 1321 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6b5196040b094455ad661e9ab9e379c7" candidate_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "bce8dbcd72a840bb8256dee0715b175f" is_pre_election: true I0916 07:55:42.197074 1249 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6b5196040b094455ad661e9ab9e379c7" candidate_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "39273ba79fdb4d31abbb34278b8b1709" is_pre_election: true I0916 07:55:42.197906 1321 raft_consensus.cc:2311] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 for term 2 because replica is either leader or believes a valid leader to be alive. I0916 07:55:42.198000 1249 raft_consensus.cc:2350] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 in term 2. I0916 07:55:42.198935 1354 leader_election.cc:310] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, 39273ba79fdb4d31abbb34278b8b1709; no voters: I0916 07:55:42.199692 1518 raft_consensus.cc:2644] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0916 07:55:42.199970 1518 raft_consensus.cc:472] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 39273ba79fdb4d31abbb34278b8b1709) I0916 07:55:42.200171 1518 raft_consensus.cc:2890] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 1 FOLLOWER]: Advancing to term 2 I0916 07:55:42.208248 1518 raft_consensus.cc:494] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } I0916 07:55:42.209599 1518 leader_election.cc:296] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 2 election: Requested vote from peers 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619), bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401) I0916 07:55:42.210201 1249 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6b5196040b094455ad661e9ab9e379c7" candidate_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "39273ba79fdb4d31abbb34278b8b1709" I0916 07:55:42.210465 1321 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6b5196040b094455ad661e9ab9e379c7" candidate_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "bce8dbcd72a840bb8256dee0715b175f" I0916 07:55:42.211108 1321 raft_consensus.cc:2311] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 for term 2 because replica is either leader or believes a valid leader to be alive. I0916 07:55:42.214598 1249 raft_consensus.cc:2350] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 in term 2. I0916 07:55:42.215445 1354 leader_election.cc:310] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, 39273ba79fdb4d31abbb34278b8b1709; no voters: bce8dbcd72a840bb8256dee0715b175f I0916 07:55:42.216490 1518 raft_consensus.cc:2644] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 2 FOLLOWER]: Leader election won for term 2 I0916 07:55:42.218365 1518 raft_consensus.cc:667] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 2 LEADER]: Becoming Leader. State: Replica: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, State: Running, Role: LEADER I0916 07:55:42.219069 1518 consensus_queue.cc:206] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } I0916 07:55:42.225351 1094 catalog_manager.cc:4120] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 reported cstate change: term changed from 1 to 2, leader changed from 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1) to 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3). New cstate: current_term: 2 leader_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } health_report { overall_health: UNKNOWN } } } I0916 07:55:42.228448 1519 raft_consensus.cc:472] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39273ba79fdb4d31abbb34278b8b1709) I0916 07:55:42.228976 1519 raft_consensus.cc:494] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } I0916 07:55:42.240803 1519 leader_election.cc:296] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619), 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:42.245553 1250 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6b5196040b094455ad661e9ab9e379c7" candidate_uuid: "bce8dbcd72a840bb8256dee0715b175f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "39273ba79fdb4d31abbb34278b8b1709" is_pre_election: true I0916 07:55:42.246399 1250 raft_consensus.cc:2280] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bce8dbcd72a840bb8256dee0715b175f in current term 2: Already voted for candidate 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 in this term. I0916 07:55:42.249402 1390 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6b5196040b094455ad661e9ab9e379c7" candidate_uuid: "bce8dbcd72a840bb8256dee0715b175f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" is_pre_election: true I0916 07:55:42.250114 1390 raft_consensus.cc:2311] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate bce8dbcd72a840bb8256dee0715b175f for term 2 because replica is either leader or believes a valid leader to be alive. I0916 07:55:42.251207 1285 leader_election.cc:310] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bce8dbcd72a840bb8256dee0715b175f; no voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, 39273ba79fdb4d31abbb34278b8b1709 I0916 07:55:42.253667 1519 raft_consensus.cc:2890] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 1 FOLLOWER]: Advancing to term 2 I0916 07:55:42.259922 1519 raft_consensus.cc:2592] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0916 07:55:42.261698 1321 tablet_service.cc:1440] Received LeaderStepDown RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" dest_uuid: "bce8dbcd72a840bb8256dee0715b175f" from {username='slave'} at 127.0.0.1:39830 I0916 07:55:42.262085 1321 raft_consensus.cc:569] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 2 LEADER]: Received request to step down I0916 07:55:42.262243 1321 raft_consensus.cc:2885] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 2 LEADER]: Stepping down as leader of term 2 I0916 07:55:42.262387 1321 raft_consensus.cc:705] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bce8dbcd72a840bb8256dee0715b175f, State: Running, Role: LEADER I0916 07:55:42.262826 1321 consensus_queue.cc:229] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:42.263597 1321 raft_consensus.cc:2890] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 2 FOLLOWER]: Advancing to term 3 I0916 07:55:42.305191 1321 raft_consensus.cc:1184] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 2 FOLLOWER]: Refusing update from remote peer 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch) I0916 07:55:42.306449 1518 consensus_queue.cc:984] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0916 07:55:42.306735 1249 raft_consensus.cc:1184] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 2 FOLLOWER]: Refusing update from remote peer 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch) I0916 07:55:42.307871 1518 consensus_queue.cc:984] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0916 07:55:42.419160 1450 raft_consensus.cc:472] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bce8dbcd72a840bb8256dee0715b175f) I0916 07:55:42.419315 1518 raft_consensus.cc:472] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bce8dbcd72a840bb8256dee0715b175f) I0916 07:55:42.419639 1450 raft_consensus.cc:494] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:42.419749 1518 raft_consensus.cc:494] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:42.421093 1518 leader_election.cc:296] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401), 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619) I0916 07:55:42.421149 1450 leader_election.cc:296] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401), 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:42.422137 1321 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "bce8dbcd72a840bb8256dee0715b175f" is_pre_election: true I0916 07:55:42.422291 1320 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "39273ba79fdb4d31abbb34278b8b1709" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "bce8dbcd72a840bb8256dee0715b175f" is_pre_election: true I0916 07:55:42.422325 1390 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "39273ba79fdb4d31abbb34278b8b1709" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" is_pre_election: true I0916 07:55:42.422793 1321 raft_consensus.cc:2350] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 in term 3. I0916 07:55:42.423074 1390 raft_consensus.cc:2350] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39273ba79fdb4d31abbb34278b8b1709 in term 2. I0916 07:55:42.423498 1320 raft_consensus.cc:2325] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39273ba79fdb4d31abbb34278b8b1709 for term 3 because replica is already servicing an update from a current leader or another vote. I0916 07:55:42.424021 1353 leader_election.cc:310] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, bce8dbcd72a840bb8256dee0715b175f; no voters: I0916 07:55:42.424300 1205 leader_election.cc:310] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, 39273ba79fdb4d31abbb34278b8b1709; no voters: I0916 07:55:42.424984 1518 raft_consensus.cc:2644] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 2 FOLLOWER]: Leader pre-election won for term 3 I0916 07:55:42.424811 1250 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "39273ba79fdb4d31abbb34278b8b1709" is_pre_election: true I0916 07:55:42.425323 1450 raft_consensus.cc:2644] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 2 FOLLOWER]: Leader pre-election won for term 3 I0916 07:55:42.425575 1518 raft_consensus.cc:472] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 2 FOLLOWER]: Starting leader election (detected failure of leader bce8dbcd72a840bb8256dee0715b175f) I0916 07:55:42.425700 1450 raft_consensus.cc:472] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 2 FOLLOWER]: Starting leader election (detected failure of leader bce8dbcd72a840bb8256dee0715b175f) I0916 07:55:42.426069 1450 raft_consensus.cc:2890] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 2 FOLLOWER]: Advancing to term 3 I0916 07:55:42.427000 1518 raft_consensus.cc:2890] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 2 FOLLOWER]: Advancing to term 3 I0916 07:55:42.434499 1450 raft_consensus.cc:494] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:42.436655 1250 raft_consensus.cc:2280] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 in current term 3: Already voted for candidate 39273ba79fdb4d31abbb34278b8b1709 in this term. I0916 07:55:42.438963 1450 leader_election.cc:296] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 3 election: Requested vote from peers bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401), 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:42.439075 1518 raft_consensus.cc:494] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:42.439477 1390 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "39273ba79fdb4d31abbb34278b8b1709" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" I0916 07:55:42.439991 1320 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "39273ba79fdb4d31abbb34278b8b1709" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "bce8dbcd72a840bb8256dee0715b175f" I0916 07:55:42.440498 1390 raft_consensus.cc:2280] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 39273ba79fdb4d31abbb34278b8b1709 in current term 3: Already voted for candidate 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 in this term. I0916 07:55:42.441553 1518 leader_election.cc:296] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 3 election: Requested vote from peers bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401), 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619) I0916 07:55:42.441474 1321 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "bce8dbcd72a840bb8256dee0715b175f" I0916 07:55:42.442368 1250 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "39273ba79fdb4d31abbb34278b8b1709" I0916 07:55:42.442837 1250 raft_consensus.cc:2280] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 in current term 3: Already voted for candidate 39273ba79fdb4d31abbb34278b8b1709 in this term. I0916 07:55:42.444612 1320 raft_consensus.cc:2350] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39273ba79fdb4d31abbb34278b8b1709 in term 3. I0916 07:55:42.445087 1321 raft_consensus.cc:2325] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 for term 3 because replica is already servicing an update from a current leader or another vote. I0916 07:55:42.445549 1202 leader_election.cc:310] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39273ba79fdb4d31abbb34278b8b1709, bce8dbcd72a840bb8256dee0715b175f; no voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 I0916 07:55:42.446270 1450 raft_consensus.cc:2644] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 3 FOLLOWER]: Leader election won for term 3 I0916 07:55:42.446483 1353 leader_election.cc:310] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90; no voters: 39273ba79fdb4d31abbb34278b8b1709, bce8dbcd72a840bb8256dee0715b175f I0916 07:55:42.447084 1518 raft_consensus.cc:2592] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority I0916 07:55:42.447342 1450 raft_consensus.cc:667] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 3 LEADER]: Becoming Leader. State: Replica: 39273ba79fdb4d31abbb34278b8b1709, State: Running, Role: LEADER I0916 07:55:42.447935 1450 consensus_queue.cc:206] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:42.452054 1090 catalog_manager.cc:4120] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 reported cstate change: term changed from 2 to 3, leader changed from bce8dbcd72a840bb8256dee0715b175f (127.0.61.2) to 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1). New cstate: current_term: 3 leader_uuid: "39273ba79fdb4d31abbb34278b8b1709" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } health_report { overall_health: UNKNOWN } } } I0916 07:55:42.471695 1321 tablet_service.cc:1440] Received LeaderStepDown RPC: tablet_id: "e5cce06208f44e4fa384db94c0b624ae" dest_uuid: "bce8dbcd72a840bb8256dee0715b175f" from {username='slave'} at 127.0.0.1:39830 I0916 07:55:42.472889 1321 raft_consensus.cc:569] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 1 LEADER]: Received request to step down I0916 07:55:42.473273 1321 raft_consensus.cc:2885] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 1 LEADER]: Stepping down as leader of term 1 I0916 07:55:42.473556 1321 raft_consensus.cc:705] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bce8dbcd72a840bb8256dee0715b175f, State: Running, Role: LEADER I0916 07:55:42.474459 1321 consensus_queue.cc:229] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:42.477165 1321 raft_consensus.cc:2890] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 1 FOLLOWER]: Advancing to term 2 I0916 07:55:42.529057 1321 raft_consensus.cc:1184] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 3 FOLLOWER]: Refusing update from remote peer 39273ba79fdb4d31abbb34278b8b1709: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch) I0916 07:55:42.530041 1450 consensus_queue.cc:984] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I0916 07:55:42.538090 1390 raft_consensus.cc:1184] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 3 FOLLOWER]: Refusing update from remote peer 39273ba79fdb4d31abbb34278b8b1709: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch) I0916 07:55:42.539443 1526 consensus_queue.cc:984] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s I0916 07:55:42.628824 1518 raft_consensus.cc:472] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bce8dbcd72a840bb8256dee0715b175f) I0916 07:55:42.628945 1525 raft_consensus.cc:472] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bce8dbcd72a840bb8256dee0715b175f) I0916 07:55:42.629405 1525 raft_consensus.cc:494] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:42.629319 1518 raft_consensus.cc:494] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:42.631311 1525 leader_election.cc:296] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401), 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:42.631688 1250 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e5cce06208f44e4fa384db94c0b624ae" candidate_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "39273ba79fdb4d31abbb34278b8b1709" is_pre_election: true I0916 07:55:42.631783 1518 leader_election.cc:296] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619), bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401) I0916 07:55:42.631855 1390 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e5cce06208f44e4fa384db94c0b624ae" candidate_uuid: "39273ba79fdb4d31abbb34278b8b1709" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" is_pre_election: true I0916 07:55:42.631886 1321 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e5cce06208f44e4fa384db94c0b624ae" candidate_uuid: "39273ba79fdb4d31abbb34278b8b1709" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "bce8dbcd72a840bb8256dee0715b175f" is_pre_election: true I0916 07:55:42.632517 1320 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e5cce06208f44e4fa384db94c0b624ae" candidate_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "bce8dbcd72a840bb8256dee0715b175f" is_pre_election: true I0916 07:55:42.632761 1390 raft_consensus.cc:2350] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39273ba79fdb4d31abbb34278b8b1709 in term 1. I0916 07:55:42.632814 1321 raft_consensus.cc:2350] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39273ba79fdb4d31abbb34278b8b1709 in term 2. I0916 07:55:42.632833 1250 raft_consensus.cc:2350] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 in term 1. I0916 07:55:42.633659 1320 raft_consensus.cc:2325] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 for term 2 because replica is already servicing an update from a current leader or another vote. I0916 07:55:42.633818 1205 leader_election.cc:310] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, 39273ba79fdb4d31abbb34278b8b1709; no voters: I0916 07:55:42.634172 1354 leader_election.cc:310] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, 39273ba79fdb4d31abbb34278b8b1709; no voters: I0916 07:55:42.634573 1525 raft_consensus.cc:2644] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0916 07:55:42.634933 1518 raft_consensus.cc:2644] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0916 07:55:42.635351 1525 raft_consensus.cc:472] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bce8dbcd72a840bb8256dee0715b175f) I0916 07:55:42.635516 1518 raft_consensus.cc:472] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bce8dbcd72a840bb8256dee0715b175f) I0916 07:55:42.635654 1525 raft_consensus.cc:2890] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 1 FOLLOWER]: Advancing to term 2 I0916 07:55:42.635848 1518 raft_consensus.cc:2890] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 1 FOLLOWER]: Advancing to term 2 I0916 07:55:42.640753 1518 raft_consensus.cc:494] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:42.640753 1525 raft_consensus.cc:494] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:42.643049 1321 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e5cce06208f44e4fa384db94c0b624ae" candidate_uuid: "39273ba79fdb4d31abbb34278b8b1709" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "bce8dbcd72a840bb8256dee0715b175f" I0916 07:55:42.644405 1250 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e5cce06208f44e4fa384db94c0b624ae" candidate_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "39273ba79fdb4d31abbb34278b8b1709" I0916 07:55:42.645100 1250 raft_consensus.cc:2280] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 in current term 2: Already voted for candidate 39273ba79fdb4d31abbb34278b8b1709 in this term. I0916 07:55:42.643081 1525 leader_election.cc:296] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 2 election: Requested vote from peers bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401), 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:42.643484 1320 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e5cce06208f44e4fa384db94c0b624ae" candidate_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "bce8dbcd72a840bb8256dee0715b175f" I0916 07:55:42.647238 1390 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e5cce06208f44e4fa384db94c0b624ae" candidate_uuid: "39273ba79fdb4d31abbb34278b8b1709" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" I0916 07:55:42.647972 1390 raft_consensus.cc:2280] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 39273ba79fdb4d31abbb34278b8b1709 in current term 2: Already voted for candidate 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 in this term. I0916 07:55:42.653682 1321 raft_consensus.cc:2350] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39273ba79fdb4d31abbb34278b8b1709 in term 2. I0916 07:55:42.654280 1320 raft_consensus.cc:2325] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 for term 2 because replica is already servicing an update from a current leader or another vote. I0916 07:55:42.642709 1518 leader_election.cc:296] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 2 election: Requested vote from peers 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619), bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401) I0916 07:55:42.656396 1202 leader_election.cc:310] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39273ba79fdb4d31abbb34278b8b1709, bce8dbcd72a840bb8256dee0715b175f; no voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 I0916 07:55:42.656584 1353 leader_election.cc:310] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90; no voters: 39273ba79fdb4d31abbb34278b8b1709, bce8dbcd72a840bb8256dee0715b175f I0916 07:55:42.657686 1518 raft_consensus.cc:2592] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority I0916 07:55:42.657721 1526 raft_consensus.cc:2644] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 2 FOLLOWER]: Leader election won for term 2 I0916 07:55:42.658401 1526 raft_consensus.cc:667] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 2 LEADER]: Becoming Leader. State: Replica: 39273ba79fdb4d31abbb34278b8b1709, State: Running, Role: LEADER I0916 07:55:42.659394 1526 consensus_queue.cc:206] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:42.663647 1094 catalog_manager.cc:4120] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 reported cstate change: term changed from 1 to 2, leader changed from bce8dbcd72a840bb8256dee0715b175f (127.0.61.2) to 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1). New cstate: current_term: 2 leader_uuid: "39273ba79fdb4d31abbb34278b8b1709" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } health_report { overall_health: UNKNOWN } } } I0916 07:55:42.699801 1390 tablet_service.cc:1440] Received LeaderStepDown RPC: tablet_id: "6b5196040b094455ad661e9ab9e379c7" dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" from {username='slave'} at 127.0.0.1:37422 I0916 07:55:42.700207 1390 raft_consensus.cc:569] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 2 LEADER]: Received request to step down I0916 07:55:42.700376 1390 raft_consensus.cc:2885] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 2 LEADER]: Stepping down as leader of term 2 I0916 07:55:42.700592 1390 raft_consensus.cc:705] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, State: Running, Role: LEADER I0916 07:55:42.701077 1390 consensus_queue.cc:229] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } I0916 07:55:42.701876 1390 raft_consensus.cc:2890] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 2 FOLLOWER]: Advancing to term 3 I0916 07:55:42.758708 1321 raft_consensus.cc:1184] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 2 FOLLOWER]: Refusing update from remote peer 39273ba79fdb4d31abbb34278b8b1709: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch) I0916 07:55:42.759820 1526 consensus_queue.cc:984] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0916 07:55:42.766671 1390 raft_consensus.cc:1184] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 2 FOLLOWER]: Refusing update from remote peer 39273ba79fdb4d31abbb34278b8b1709: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch) I0916 07:55:42.767956 1526 consensus_queue.cc:984] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0916 07:55:42.829928 1519 raft_consensus.cc:472] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90) I0916 07:55:42.830338 1519 raft_consensus.cc:494] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } I0916 07:55:42.831523 1519 leader_election.cc:296] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619), 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:42.832403 1250 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6b5196040b094455ad661e9ab9e379c7" candidate_uuid: "bce8dbcd72a840bb8256dee0715b175f" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "39273ba79fdb4d31abbb34278b8b1709" is_pre_election: true I0916 07:55:42.832403 1390 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6b5196040b094455ad661e9ab9e379c7" candidate_uuid: "bce8dbcd72a840bb8256dee0715b175f" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" is_pre_election: true I0916 07:55:42.833372 1250 raft_consensus.cc:2311] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bce8dbcd72a840bb8256dee0715b175f for term 3 because replica is either leader or believes a valid leader to be alive. I0916 07:55:42.833333 1390 raft_consensus.cc:2350] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bce8dbcd72a840bb8256dee0715b175f in term 3. I0916 07:55:42.834357 1285 leader_election.cc:310] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, bce8dbcd72a840bb8256dee0715b175f; no voters: I0916 07:55:42.835016 1519 raft_consensus.cc:2644] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 2 FOLLOWER]: Leader pre-election won for term 3 I0916 07:55:42.835263 1519 raft_consensus.cc:472] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90) I0916 07:55:42.835471 1519 raft_consensus.cc:2890] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 2 FOLLOWER]: Advancing to term 3 I0916 07:55:42.839026 1519 raft_consensus.cc:494] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } I0916 07:55:42.840173 1519 leader_election.cc:296] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 3 election: Requested vote from peers 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619), 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:42.841018 1250 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6b5196040b094455ad661e9ab9e379c7" candidate_uuid: "bce8dbcd72a840bb8256dee0715b175f" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "39273ba79fdb4d31abbb34278b8b1709" I0916 07:55:42.841018 1390 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6b5196040b094455ad661e9ab9e379c7" candidate_uuid: "bce8dbcd72a840bb8256dee0715b175f" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" I0916 07:55:42.841840 1250 raft_consensus.cc:2311] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate bce8dbcd72a840bb8256dee0715b175f for term 3 because replica is either leader or believes a valid leader to be alive. I0916 07:55:42.845221 1390 raft_consensus.cc:2350] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bce8dbcd72a840bb8256dee0715b175f in term 3. I0916 07:55:42.846101 1285 leader_election.cc:310] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, bce8dbcd72a840bb8256dee0715b175f; no voters: 39273ba79fdb4d31abbb34278b8b1709 I0916 07:55:42.846681 1519 raft_consensus.cc:2644] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 3 FOLLOWER]: Leader election won for term 3 I0916 07:55:42.846990 1519 raft_consensus.cc:667] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 3 LEADER]: Becoming Leader. State: Replica: bce8dbcd72a840bb8256dee0715b175f, State: Running, Role: LEADER I0916 07:55:42.847515 1519 consensus_queue.cc:206] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } I0916 07:55:42.851423 1526 raft_consensus.cc:472] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90) I0916 07:55:42.851848 1526 raft_consensus.cc:494] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } I0916 07:55:42.852146 1094 catalog_manager.cc:4120] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f reported cstate change: term changed from 2 to 3, leader changed from 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3) to bce8dbcd72a840bb8256dee0715b175f (127.0.61.2). New cstate: current_term: 3 leader_uuid: "bce8dbcd72a840bb8256dee0715b175f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } health_report { overall_health: HEALTHY } } } I0916 07:55:42.869498 1321 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6b5196040b094455ad661e9ab9e379c7" candidate_uuid: "39273ba79fdb4d31abbb34278b8b1709" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "bce8dbcd72a840bb8256dee0715b175f" is_pre_election: true I0916 07:55:42.870211 1321 raft_consensus.cc:2311] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate 39273ba79fdb4d31abbb34278b8b1709 for term 3 because replica is either leader or believes a valid leader to be alive. I0916 07:55:42.871381 1526 leader_election.cc:296] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451), bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401) I0916 07:55:42.871222 1390 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6b5196040b094455ad661e9ab9e379c7" candidate_uuid: "39273ba79fdb4d31abbb34278b8b1709" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" is_pre_election: true I0916 07:55:42.872058 1390 raft_consensus.cc:2280] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39273ba79fdb4d31abbb34278b8b1709 in current term 3: Already voted for candidate bce8dbcd72a840bb8256dee0715b175f in this term. I0916 07:55:42.875154 1205 leader_election.cc:310] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39273ba79fdb4d31abbb34278b8b1709; no voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, bce8dbcd72a840bb8256dee0715b175f I0916 07:55:42.876233 1530 raft_consensus.cc:2890] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 2 FOLLOWER]: Advancing to term 3 I0916 07:55:42.880662 1530 raft_consensus.cc:2592] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0916 07:55:42.910377 1094 catalog_manager.cc:2365] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42786: table { table_id: "074536fca6144499906b7c13e035c370" table_name: "test-table" } new_table_name: "NewTableName" I0916 07:55:42.932301 1390 raft_consensus.cc:1184] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 3 FOLLOWER]: Refusing update from remote peer bce8dbcd72a840bb8256dee0715b175f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch) I0916 07:55:42.932602 244 master.cc:247] Master@127.0.61.61:37305 shutting down... I0916 07:55:42.934231 1250 raft_consensus.cc:1184] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 3 FOLLOWER]: Refusing update from remote peer bce8dbcd72a840bb8256dee0715b175f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch) I0916 07:55:42.934381 1520 consensus_queue.cc:984] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I0916 07:55:42.936223 1520 consensus_queue.cc:984] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [LEADER]: Connected to new peer: Peer: permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I0916 07:55:42.949246 1075 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0916 07:55:42.953634 1076 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0916 07:55:42.961269 1076 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0916 07:55:43.015933 1077 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0916 07:55:43.018374 1077 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0916 07:55:43.026134 1535 tablet.cc:1223] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f: Alter schema from ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0916 07:55:43.029783 1534 tablet.cc:1223] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709: Alter schema from ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0916 07:55:43.038954 1538 tablet.cc:1223] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f: Alter schema from ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0916 07:55:43.043540 1533 tablet.cc:1223] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f: Alter schema from ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0916 07:55:43.043951 1536 tablet.cc:1223] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709: Alter schema from ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0916 07:55:43.062299 1077 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0916 07:55:43.062661 1537 tablet.cc:1223] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709: Alter schema from ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0916 07:55:43.066457 1076 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0916 07:55:43.069983 1075 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0916 07:55:43.072023 1540 tablet.cc:1223] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Alter schema from ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0916 07:55:43.093181 1077 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0916 07:55:43.096797 1076 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0916 07:55:43.096760 1541 tablet.cc:1223] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Alter schema from ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0916 07:55:43.100683 1076 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0916 07:55:43.102954 1542 tablet.cc:1223] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Alter schema from ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0916 07:55:43.106812 1077 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0916 07:55:43.127094 1075 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0916 07:55:43.127338 1077 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0916 07:55:43.132037 1075 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0916 07:55:43.132961 1076 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0916 07:55:43.143451 244 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 1 LEADER]: Raft consensus shutting down. I0916 07:55:43.143932 1075 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0916 07:55:43.144323 244 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 1 FOLLOWER]: Raft consensus is shut down! I0916 07:55:43.144750 244 tablet_replica.cc:273] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416: stopping tablet replica I0916 07:55:43.164786 244 master.cc:258] Master@127.0.61.61:37305 shutdown complete. I0916 07:55:43.312338 1546 raft_consensus.cc:472] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5fe1f662b7a54c0f823ca2d656131416) I0916 07:55:43.312800 1546 raft_consensus.cc:494] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } I0916 07:55:43.314057 1546 leader_election.cc:296] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd1e33dc408e40c7a8bae2850d5a6b6f (127.0.61.62:33603), 5fe1f662b7a54c0f823ca2d656131416 (127.0.61.61:37305) I0916 07:55:43.316154 1047 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d933e518c06d4c7d8b4f743167c2c643" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" is_pre_election: true I0916 07:55:43.316879 1047 raft_consensus.cc:2350] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d933e518c06d4c7d8b4f743167c2c643 in term 1. W0916 07:55:43.316884 1134 leader_election.cc:341] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5fe1f662b7a54c0f823ca2d656131416 (127.0.61.61:37305): Network error: Client connection negotiation failed: client connection to 127.0.61.61:37305: connect: Connection refused (error 111) I0916 07:55:43.317855 1547 raft_consensus.cc:472] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5fe1f662b7a54c0f823ca2d656131416) I0916 07:55:43.318084 1135 leader_election.cc:310] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d933e518c06d4c7d8b4f743167c2c643, fd1e33dc408e40c7a8bae2850d5a6b6f; no voters: 5fe1f662b7a54c0f823ca2d656131416 I0916 07:55:43.318183 1547 raft_consensus.cc:494] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } I0916 07:55:43.319000 1546 raft_consensus.cc:2644] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0916 07:55:43.319303 1546 raft_consensus.cc:472] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5fe1f662b7a54c0f823ca2d656131416) I0916 07:55:43.319561 1546 raft_consensus.cc:2890] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 1 FOLLOWER]: Advancing to term 2 I0916 07:55:43.321306 1547 leader_election.cc:296] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5fe1f662b7a54c0f823ca2d656131416 (127.0.61.61:37305), d933e518c06d4c7d8b4f743167c2c643 (127.0.61.60:37785) I0916 07:55:43.322182 1160 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "d933e518c06d4c7d8b4f743167c2c643" is_pre_election: true W0916 07:55:43.323065 1020 leader_election.cc:341] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5fe1f662b7a54c0f823ca2d656131416 (127.0.61.61:37305): Network error: Client connection negotiation failed: client connection to 127.0.61.61:37305: connect: Connection refused (error 111) I0916 07:55:43.325134 1546 raft_consensus.cc:494] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } I0916 07:55:43.326073 1160 raft_consensus.cc:2280] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fd1e33dc408e40c7a8bae2850d5a6b6f in current term 2: Already voted for candidate d933e518c06d4c7d8b4f743167c2c643 in this term. I0916 07:55:43.326581 1546 leader_election.cc:296] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [CANDIDATE]: Term 2 election: Requested vote from peers fd1e33dc408e40c7a8bae2850d5a6b6f (127.0.61.62:33603), 5fe1f662b7a54c0f823ca2d656131416 (127.0.61.61:37305) I0916 07:55:43.327240 1047 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d933e518c06d4c7d8b4f743167c2c643" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" I0916 07:55:43.327497 1019 leader_election.cc:310] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd1e33dc408e40c7a8bae2850d5a6b6f; no voters: 5fe1f662b7a54c0f823ca2d656131416, d933e518c06d4c7d8b4f743167c2c643 I0916 07:55:43.327874 1047 raft_consensus.cc:2890] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 1 FOLLOWER]: Advancing to term 2 W0916 07:55:43.328269 1134 leader_election.cc:341] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5fe1f662b7a54c0f823ca2d656131416 (127.0.61.61:37305): Network error: Client connection negotiation failed: client connection to 127.0.61.61:37305: connect: Connection refused (error 111) I0916 07:55:43.331660 1047 raft_consensus.cc:2350] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d933e518c06d4c7d8b4f743167c2c643 in term 2. I0916 07:55:43.332036 1547 raft_consensus.cc:2592] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0916 07:55:43.332504 1135 leader_election.cc:310] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d933e518c06d4c7d8b4f743167c2c643, fd1e33dc408e40c7a8bae2850d5a6b6f; no voters: 5fe1f662b7a54c0f823ca2d656131416 I0916 07:55:43.333109 1546 raft_consensus.cc:2644] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 2 FOLLOWER]: Leader election won for term 2 I0916 07:55:43.334600 1546 raft_consensus.cc:667] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 2 LEADER]: Becoming Leader. State: Replica: d933e518c06d4c7d8b4f743167c2c643, State: Running, Role: LEADER I0916 07:55:43.335203 1546 consensus_queue.cc:206] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 14, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } I0916 07:55:43.338737 1550 sys_catalog.cc:340] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d933e518c06d4c7d8b4f743167c2c643. Latest consensus state: current_term: 2 leader_uuid: "d933e518c06d4c7d8b4f743167c2c643" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:43.339690 1550 sys_catalog.cc:343] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [sys.catalog]: This master's current role is: LEADER I0916 07:55:43.341357 1551 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0916 07:55:43.345227 1551 catalog_manager.cc:387] Loaded metadata for table NewTableName [id=074536fca6144499906b7c13e035c370] I0916 07:55:43.347920 1551 catalog_manager.cc:441] Loaded metadata for tablet 6b5196040b094455ad661e9ab9e379c7 (table NewTableName [id=074536fca6144499906b7c13e035c370]) I0916 07:55:43.348723 1551 catalog_manager.cc:441] Loaded metadata for tablet e02eeaf40c1d4a299b70f4a8f1514e31 (table NewTableName [id=074536fca6144499906b7c13e035c370]) I0916 07:55:43.349658 1551 catalog_manager.cc:441] Loaded metadata for tablet e5cce06208f44e4fa384db94c0b624ae (table NewTableName [id=074536fca6144499906b7c13e035c370]) I0916 07:55:43.350428 1551 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0916 07:55:43.353683 1551 catalog_manager.cc:1106] Loading token signing keys... I0916 07:55:43.355890 1551 catalog_manager.cc:4446] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643: Loaded TSK: 0 I0916 07:55:43.387964 244 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 47516us I0916 07:55:43.388387 244 fs_manager.cc:263] Metadata directory not provided I0916 07:55:43.388550 244 fs_manager.cc:266] Using existing metadata directory in first data directory I0916 07:55:43.391105 244 fs_manager.cc:399] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s I0916 07:55:43.392194 244 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0916 07:55:43.400789 244 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0916 07:55:43.401002 244 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/master-1-root uuid: "5fe1f662b7a54c0f823ca2d656131416" format_stamp: "Formatted at 2019-09-16 07:55:39 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:43.401309 244 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/master-1-root metadata directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/master-1-root 1 data directories: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/master-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0916 07:55:43.421764 1134 consensus_peers.cc:458] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 -> Peer 5fe1f662b7a54c0f823ca2d656131416 (127.0.61.61:37305): Couldn't send request to peer 5fe1f662b7a54c0f823ca2d656131416. Status: Network error: Client connection negotiation failed: client connection to 127.0.61.61:37305: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0916 07:55:43.455139 1047 raft_consensus.cc:1184] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 2 FOLLOWER]: Refusing update from remote peer d933e518c06d4c7d8b4f743167c2c643: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch) I0916 07:55:43.456241 1550 consensus_queue.cc:984] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s I0916 07:55:43.458827 1547 sys_catalog.cc:340] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d933e518c06d4c7d8b4f743167c2c643. Latest consensus state: current_term: 2 leader_uuid: "d933e518c06d4c7d8b4f743167c2c643" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:43.459527 1547 sys_catalog.cc:343] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [sys.catalog]: This master's current role is: FOLLOWER I0916 07:55:43.461679 1546 sys_catalog.cc:340] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "d933e518c06d4c7d8b4f743167c2c643" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:43.462539 1546 sys_catalog.cc:343] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [sys.catalog]: This master's current role is: LEADER I0916 07:55:43.465312 1547 sys_catalog.cc:340] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "d933e518c06d4c7d8b4f743167c2c643" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:43.466015 1547 sys_catalog.cc:343] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [sys.catalog]: This master's current role is: FOLLOWER I0916 07:55:43.515604 244 rpc_server.cc:205] RPC server started. Bound to: 127.0.61.61:37305 I0916 07:55:43.517833 244 webserver.cc:374] Webserver started at http://127.0.61.61:38341/ using document root and password file I0916 07:55:43.524369 1608 sys_catalog.cc:188] Verifying existing consensus state I0916 07:55:43.527978 1608 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416: Bootstrap starting. I0916 07:55:43.542503 1344 heartbeater.cc:447] Master 127.0.61.60:37785 was elected leader, sending a full tablet report... I0916 07:55:43.544132 1273 heartbeater.cc:447] Master 127.0.61.60:37785 was elected leader, sending a full tablet report... I0916 07:55:43.554677 1415 heartbeater.cc:447] Master 127.0.61.60:37785 was elected leader, sending a full tablet report... I0916 07:55:43.589802 1149 catalog_manager.cc:4580] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643: NewTableName [id=074536fca6144499906b7c13e035c370] alter complete (version 1) I0916 07:55:43.599375 1345 heartbeater.cc:322] Connected to a master server at 127.0.61.61:37305 I0916 07:55:43.607246 1608 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=6 ignored=0} mutations{seen=12 ignored=0} orphaned_commits=0. Pending: 0 replicates I0916 07:55:43.614037 1608 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416: Bootstrap complete. I0916 07:55:43.620963 1274 heartbeater.cc:322] Connected to a master server at 127.0.61.61:37305 I0916 07:55:43.623842 1416 heartbeater.cc:322] Connected to a master server at 127.0.61.61:37305 I0916 07:55:43.633810 244 master.cc:247] Master@127.0.61.60:37785 shutting down... I0916 07:55:43.640813 1608 raft_consensus.cc:340] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } I0916 07:55:43.647662 1608 raft_consensus.cc:705] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fe1f662b7a54c0f823ca2d656131416, State: Initialized, Role: FOLLOWER I0916 07:55:43.653605 1133 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0916 07:55:43.710372 1608 consensus_queue.cc:229] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 1.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } I0916 07:55:43.714685 1133 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0916 07:55:43.727550 1133 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0916 07:55:43.733109 1135 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0916 07:55:43.732586 1608 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:43.734283 1608 sys_catalog.cc:343] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: This master's current role is: FOLLOWER I0916 07:55:43.737118 1617 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:43.739521 1617 sys_catalog.cc:343] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: This master's current role is: FOLLOWER I0916 07:55:43.740782 1585 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 1 FOLLOWER]: Advancing to term 2 I0916 07:55:43.740427 1608 sys_catalog.cc:428] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: configured and running, proceeding with master startup. I0916 07:55:43.758211 244 raft_consensus.cc:2147] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 2 LEADER]: Raft consensus shutting down. I0916 07:55:43.758843 244 raft_consensus.cc:2174] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 2 FOLLOWER]: Raft consensus is shut down! I0916 07:55:43.759188 244 tablet_replica.cc:273] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643: stopping tablet replica I0916 07:55:43.760762 1617 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d933e518c06d4c7d8b4f743167c2c643. Latest consensus state: current_term: 2 leader_uuid: "d933e518c06d4c7d8b4f743167c2c643" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:43.761941 1617 sys_catalog.cc:343] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: This master's current role is: FOLLOWER I0916 07:55:43.763931 1575 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" instance_seqno: 1568620540262017) as {username='slave'} at 127.0.0.1:42826; Asking this server to re-register. I0916 07:55:43.766593 1416 heartbeater.cc:406] Registering TS with master... I0916 07:55:43.768673 1619 catalog_manager.cc:1142] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416: acquiring CA information for follower catalog manager: success I0916 07:55:43.769073 1575 ts_manager.cc:161] Registered new tserver with Master: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:43.764633 1622 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "d933e518c06d4c7d8b4f743167c2c643" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:43.769793 1622 sys_catalog.cc:343] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: This master's current role is: FOLLOWER I0916 07:55:43.774718 1619 catalog_manager.cc:1170] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0916 07:55:43.783994 244 master.cc:258] Master@127.0.61.60:37785 shutdown complete. I0916 07:55:43.831336 1575 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" instance_seqno: 1568620539861819) as {username='slave'} at 127.0.0.1:42824; Asking this server to re-register. I0916 07:55:43.832439 1274 heartbeater.cc:406] Registering TS with master... I0916 07:55:43.835005 1575 ts_manager.cc:161] Registered new tserver with Master: 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619) I0916 07:55:43.836235 1574 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" instance_seqno: 1568620540049147) as {username='slave'} at 127.0.0.1:42822; Asking this server to re-register. I0916 07:55:43.837685 1345 heartbeater.cc:406] Registering TS with master... I0916 07:55:43.839761 1574 ts_manager.cc:161] Registered new tserver with Master: bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401) I0916 07:55:43.915010 1547 raft_consensus.cc:472] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d933e518c06d4c7d8b4f743167c2c643) I0916 07:55:43.915410 1547 raft_consensus.cc:494] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } I0916 07:55:43.916860 1547 leader_election.cc:296] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5fe1f662b7a54c0f823ca2d656131416 (127.0.61.61:37305), d933e518c06d4c7d8b4f743167c2c643 (127.0.61.60:37785) W0916 07:55:43.924809 1019 leader_election.cc:341] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d933e518c06d4c7d8b4f743167c2c643 (127.0.61.60:37785): Network error: Client connection negotiation failed: client connection to 127.0.61.60:37785: connect: Connection refused (error 111) I0916 07:55:43.932971 1585 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "5fe1f662b7a54c0f823ca2d656131416" is_pre_election: true I0916 07:55:43.933619 1585 raft_consensus.cc:2311] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fd1e33dc408e40c7a8bae2850d5a6b6f for term 3 because replica is either leader or believes a valid leader to be alive. I0916 07:55:43.934481 1020 leader_election.cc:310] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd1e33dc408e40c7a8bae2850d5a6b6f; no voters: 5fe1f662b7a54c0f823ca2d656131416, d933e518c06d4c7d8b4f743167c2c643 I0916 07:55:43.935427 1547 raft_consensus.cc:2592] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0916 07:55:43.950459 1622 raft_consensus.cc:472] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d933e518c06d4c7d8b4f743167c2c643) I0916 07:55:43.950877 1622 raft_consensus.cc:494] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } I0916 07:55:43.952177 1622 leader_election.cc:296] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fd1e33dc408e40c7a8bae2850d5a6b6f (127.0.61.62:33603), d933e518c06d4c7d8b4f743167c2c643 (127.0.61.60:37785) W0916 07:55:43.959514 1555 leader_election.cc:341] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d933e518c06d4c7d8b4f743167c2c643 (127.0.61.60:37785): Network error: Client connection negotiation failed: client connection to 127.0.61.60:37785: connect: Connection refused (error 111) I0916 07:55:43.964215 1047 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5fe1f662b7a54c0f823ca2d656131416" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" is_pre_election: true I0916 07:55:43.964764 1047 raft_consensus.cc:2350] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fe1f662b7a54c0f823ca2d656131416 in term 2. I0916 07:55:43.965505 1557 leader_election.cc:310] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5fe1f662b7a54c0f823ca2d656131416, fd1e33dc408e40c7a8bae2850d5a6b6f; no voters: d933e518c06d4c7d8b4f743167c2c643 I0916 07:55:43.966135 1622 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 2 FOLLOWER]: Leader pre-election won for term 3 I0916 07:55:43.966346 1622 raft_consensus.cc:472] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d933e518c06d4c7d8b4f743167c2c643) I0916 07:55:43.966540 1622 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 2 FOLLOWER]: Advancing to term 3 I0916 07:55:43.970239 1622 raft_consensus.cc:494] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } I0916 07:55:43.971405 1622 leader_election.cc:296] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [CANDIDATE]: Term 3 election: Requested vote from peers fd1e33dc408e40c7a8bae2850d5a6b6f (127.0.61.62:33603), d933e518c06d4c7d8b4f743167c2c643 (127.0.61.60:37785) I0916 07:55:43.972095 1047 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5fe1f662b7a54c0f823ca2d656131416" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" W0916 07:55:43.972498 1555 leader_election.cc:341] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d933e518c06d4c7d8b4f743167c2c643 (127.0.61.60:37785): Network error: Client connection negotiation failed: client connection to 127.0.61.60:37785: connect: Connection refused (error 111) I0916 07:55:43.972548 1047 raft_consensus.cc:2890] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 2 FOLLOWER]: Advancing to term 3 I0916 07:55:43.976989 1047 raft_consensus.cc:2350] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fe1f662b7a54c0f823ca2d656131416 in term 3. I0916 07:55:43.977699 1557 leader_election.cc:310] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5fe1f662b7a54c0f823ca2d656131416, fd1e33dc408e40c7a8bae2850d5a6b6f; no voters: d933e518c06d4c7d8b4f743167c2c643 I0916 07:55:43.978265 1622 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 3 FOLLOWER]: Leader election won for term 3 I0916 07:55:43.978569 1622 raft_consensus.cc:667] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 3 LEADER]: Becoming Leader. State: Replica: 5fe1f662b7a54c0f823ca2d656131416, State: Running, Role: LEADER I0916 07:55:43.978927 1622 consensus_queue.cc:206] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 2.16, Last appended by leader: 16, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } I0916 07:55:43.980705 1617 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fe1f662b7a54c0f823ca2d656131416. Latest consensus state: current_term: 3 leader_uuid: "5fe1f662b7a54c0f823ca2d656131416" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:43.981197 1617 sys_catalog.cc:343] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: This master's current role is: LEADER I0916 07:55:43.982450 1629 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0916 07:55:43.985347 1629 catalog_manager.cc:387] Loaded metadata for table NewTableName [id=074536fca6144499906b7c13e035c370] I0916 07:55:43.988124 1629 catalog_manager.cc:441] Loaded metadata for tablet 6b5196040b094455ad661e9ab9e379c7 (table NewTableName [id=074536fca6144499906b7c13e035c370]) I0916 07:55:43.988811 1629 catalog_manager.cc:441] Loaded metadata for tablet e02eeaf40c1d4a299b70f4a8f1514e31 (table NewTableName [id=074536fca6144499906b7c13e035c370]) I0916 07:55:43.989418 1629 catalog_manager.cc:441] Loaded metadata for tablet e5cce06208f44e4fa384db94c0b624ae (table NewTableName [id=074536fca6144499906b7c13e035c370]) I0916 07:55:43.989915 1629 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0916 07:55:43.992727 1629 catalog_manager.cc:1106] Loading token signing keys... I0916 07:55:43.995673 1629 catalog_manager.cc:4446] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416: Loaded TSK: 0 I0916 07:55:44.008603 244 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 48016us I0916 07:55:44.008960 244 fs_manager.cc:263] Metadata directory not provided I0916 07:55:44.009088 244 fs_manager.cc:266] Using existing metadata directory in first data directory I0916 07:55:44.011821 244 fs_manager.cc:399] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s I0916 07:55:44.012941 244 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0916 07:55:44.023082 244 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s I0916 07:55:44.023272 244 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/master-2-root uuid: "d933e518c06d4c7d8b4f743167c2c643" format_stamp: "Formatted at 2019-09-16 07:55:39 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:44.023495 244 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/master-2-root metadata directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/master-2-root 1 data directories: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/master-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0916 07:55:44.074414 1047 raft_consensus.cc:1184] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 3 FOLLOWER]: Refusing update from remote peer 5fe1f662b7a54c0f823ca2d656131416: Log matching property violated. Preceding OpId in replica: term: 2 index: 16. Preceding OpId from leader: term: 3 index: 17. (index mismatch) I0916 07:55:44.075899 1617 consensus_queue.cc:984] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s W0916 07:55:44.079823 1555 consensus_peers.cc:458] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 -> Peer d933e518c06d4c7d8b4f743167c2c643 (127.0.61.60:37785): Couldn't send request to peer d933e518c06d4c7d8b4f743167c2c643. Status: Network error: Client connection negotiation failed: client connection to 127.0.61.60:37785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0916 07:55:44.080116 1547 sys_catalog.cc:340] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fe1f662b7a54c0f823ca2d656131416. Latest consensus state: current_term: 3 leader_uuid: "5fe1f662b7a54c0f823ca2d656131416" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:44.080972 1547 sys_catalog.cc:343] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [sys.catalog]: This master's current role is: FOLLOWER I0916 07:55:44.082214 244 rpc_server.cc:205] RPC server started. Bound to: 127.0.61.60:37785 I0916 07:55:44.085058 1617 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "5fe1f662b7a54c0f823ca2d656131416" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:44.085902 1617 sys_catalog.cc:343] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: This master's current role is: LEADER I0916 07:55:44.087242 244 webserver.cc:374] Webserver started at http://127.0.61.60:45039/ using document root and password file I0916 07:55:44.088086 1547 sys_catalog.cc:340] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "5fe1f662b7a54c0f823ca2d656131416" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:44.088898 1547 sys_catalog.cc:343] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [sys.catalog]: This master's current role is: FOLLOWER I0916 07:55:44.093084 1685 sys_catalog.cc:188] Verifying existing consensus state I0916 07:55:44.096859 1685 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643: Bootstrap starting. I0916 07:55:44.130540 1685 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=6 ignored=0} mutations{seen=13 ignored=0} orphaned_commits=0. Pending: 0 replicates I0916 07:55:44.131358 1685 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643: Bootstrap complete. I0916 07:55:44.133347 1685 raft_consensus.cc:340] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } I0916 07:55:44.134124 1685 raft_consensus.cc:705] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d933e518c06d4c7d8b4f743167c2c643, State: Initialized, Role: FOLLOWER I0916 07:55:44.134670 1685 consensus_queue.cc:229] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 2.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } I0916 07:55:44.138887 1415 heartbeater.cc:322] Connected to a master server at 127.0.61.60:37785 I0916 07:55:44.142117 1274 heartbeater.cc:447] Master 127.0.61.61:37305 was elected leader, sending a full tablet report... I0916 07:55:44.146600 1345 heartbeater.cc:447] Master 127.0.61.61:37305 was elected leader, sending a full tablet report... I0916 07:55:44.157610 1688 sys_catalog.cc:340] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:44.158571 1688 sys_catalog.cc:343] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [sys.catalog]: This master's current role is: FOLLOWER I0916 07:55:44.161629 1344 heartbeater.cc:322] Connected to a master server at 127.0.61.60:37785 I0916 07:55:44.137580 1685 sys_catalog.cc:340] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:44.165091 1685 sys_catalog.cc:343] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [sys.catalog]: This master's current role is: FOLLOWER I0916 07:55:44.165858 1685 sys_catalog.cc:428] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [sys.catalog]: configured and running, proceeding with master startup. I0916 07:55:44.171123 1273 heartbeater.cc:322] Connected to a master server at 127.0.61.60:37785 I0916 07:55:44.172688 1690 catalog_manager.cc:1142] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643: acquiring CA information for follower catalog manager: success I0916 07:55:44.173818 1651 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" instance_seqno: 1568620539861819) as {username='slave'} at 127.0.0.1:41138; Asking this server to re-register. I0916 07:55:44.174962 1273 heartbeater.cc:406] Registering TS with master... I0916 07:55:44.176183 1690 catalog_manager.cc:1170] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0916 07:55:44.177136 1651 ts_manager.cc:161] Registered new tserver with Master: 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619) I0916 07:55:44.179664 1416 heartbeater.cc:447] Master 127.0.61.61:37305 was elected leader, sending a full tablet report... I0916 07:55:44.194516 1651 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" instance_seqno: 1568620540262017) as {username='slave'} at 127.0.0.1:41134; Asking this server to re-register. I0916 07:55:44.195816 1415 heartbeater.cc:406] Registering TS with master... I0916 07:55:44.196027 244 master.cc:247] Master@127.0.61.61:37305 shutting down... I0916 07:55:44.199313 1650 ts_manager.cc:161] Registered new tserver with Master: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:44.206218 1651 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" instance_seqno: 1568620540049147) as {username='slave'} at 127.0.0.1:41136; Asking this server to re-register. I0916 07:55:44.207547 1344 heartbeater.cc:406] Registering TS with master... I0916 07:55:44.210506 1649 ts_manager.cc:161] Registered new tserver with Master: bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401) I0916 07:55:44.220113 1661 raft_consensus.cc:2890] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 2 FOLLOWER]: Advancing to term 3 I0916 07:55:44.233041 1693 sys_catalog.cc:340] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "5fe1f662b7a54c0f823ca2d656131416" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:44.234020 244 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 3 LEADER]: Raft consensus shutting down. I0916 07:55:44.234047 1693 sys_catalog.cc:343] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [sys.catalog]: This master's current role is: FOLLOWER I0916 07:55:44.233502 1688 sys_catalog.cc:340] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fe1f662b7a54c0f823ca2d656131416. Latest consensus state: current_term: 3 leader_uuid: "5fe1f662b7a54c0f823ca2d656131416" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } W0916 07:55:44.234524 1617 consensus_queue.cc:1099] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "d933e518c06d4c7d8b4f743167c2c643" responder_term: 3 status { last_received { term: 3 index: 17 } last_committed_idx: 17 last_received_current_leader { term: 3 index: 17 } } I0916 07:55:44.234933 1688 sys_catalog.cc:343] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [sys.catalog]: This master's current role is: FOLLOWER I0916 07:55:44.235647 244 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 3 FOLLOWER]: Raft consensus is shut down! I0916 07:55:44.236296 244 tablet_replica.cc:273] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416: stopping tablet replica I0916 07:55:44.248971 244 master.cc:258] Master@127.0.61.61:37305 shutdown complete. ================== WARNING: ThreadSanitizer: data race (pid=244) Write of size 8 at 0x7b70000d0468 by main thread: #0 pthread_cond_destroy /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1199 (ts_tablet_manager-itest+0x4740dc) #1 kudu::ConditionVariable::~ConditionVariable() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/condition_variable.cc:46:12 (libkudu_util.so+0x112b8e) #2 kudu::CountDownLatch::~CountDownLatch() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/countdown_latch.h:31:7 (libitest_util.so+0x8078a) #3 kudu::Promise::~Promise() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/promise.h:34:15 (libmaster.so+0x185845) #4 kudu::master::Master::~Master() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/master/master.cc:128:1 (libmaster.so+0x182c42) #5 kudu::master::Master::~Master() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/master/master.cc:126:19 (libmaster.so+0x182e99) #6 std::__1::default_delete::operator()(kudu::master::Master*) const /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/c++/v1/memory:2285:5 (libmaster.so+0x1a3093) #7 std::__1::unique_ptr >::reset(kudu::master::Master*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/c++/v1/memory:2598 (libmaster.so+0x1a3093) #8 kudu::master::MiniMaster::Shutdown() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/master/mini_master.cc:118 (libmaster.so+0x1a3093) #9 kudu::tserver::TsTabletManagerITest_TestTableStats_Test::TestBody() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/ts_tablet_manager-itest.cc:735:20 (ts_tablet_manager-itest+0x4e4a4d) #10 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x552ef) #11 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x552ef) #12 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474:5 (libgmock.so+0x344b8) #13 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x3574c) #14 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x36226) #15 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x425fa) #16 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x5625f) #17 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x5625f) #18 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x41ee2) #19 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x351b) #20 main /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2cc6) Previous read of size 8 at 0x7b70000d0468 by thread T264 (mutexes: write M35153): #0 pthread_cond_broadcast /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1192 (ts_tablet_manager-itest+0x46cd00) #1 kudu::ConditionVariable::Broadcast() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/condition_variable.cc:133:12 (libkudu_util.so+0x112f9e) #2 kudu::CountDownLatch::CountDown(int) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/countdown_latch.h:57:13 (libitest_util.so+0x7f2cd) #3 kudu::CountDownLatch::CountDown() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/countdown_latch.h:65:5 (libitest_util.so+0x7f36e) #4 kudu::Promise::Set(kudu::Status const&) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/promise.h:69:12 (libmaster.so+0x185dcb) #5 kudu::master::Master::InitCatalogManagerTask() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/master/master.cc:207:16 (libmaster.so+0x183f52) #6 kudu::internal::RunnableAdapter::Run(kudu::master::Master*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/bind_internal.h:136:12 (libmaster.so+0x188489) #7 kudu::internal::InvokeHelper, void (kudu::master::Master*)>::MakeItSo(kudu::internal::RunnableAdapter, kudu::master::Master*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/bind_internal.h:873:14 (libmaster.so+0x1883a5) #8 kudu::internal::Invoker<1, kudu::internal::BindState, void (kudu::master::Master*), void (kudu::internal::UnretainedWrapper)>, void (kudu::master::Master*)>::Run(kudu::internal::BindStateBase*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/bind_internal.h:1065:12 (libmaster.so+0x1882ea) #9 kudu::Callback::Run() const /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/callback.h:396:12 (libconsensus.so+0xa97fd) #10 kudu::ClosureRunnable::Run() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:76:9 (libkudu_util.so+0x1fd94d) #11 kudu::ThreadPool::DispatchThread() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:686:22 (libkudu_util.so+0x1f9678) #12 boost::_mfi::mf0::operator()(kudu::ThreadPool*) const /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkudu_util.so+0x2045e9) #13 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkudu_util.so+0x20453a) #14 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkudu_util.so+0x2044c3) #15 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkudu_util.so+0x2042b9) #16 boost::function0::operator()() const /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xba0b1) #17 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.cc:657:3 (libkudu_util.so+0x1ee174) As if synchronized via sleep: #0 nanosleep /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:355 (ts_tablet_manager-itest+0x450b0a) #1 base::SleepForNanoseconds(long) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/sysinfo.cc:95:10 (libgutil.so+0x61d02) #2 kudu::SleepFor(kudu::MonoDelta const&) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/monotime.cc:267:3 (libkudu_util.so+0x1b5826) #3 kudu::tserver::TsTabletManagerITest_TestTableStats_Test::TestBody() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/ts_tablet_manager-itest.cc:741:7 (ts_tablet_manager-itest+0x4e4b0a) #4 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x552ef) #5 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x552ef) #6 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474:5 (libgmock.so+0x344b8) #7 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x3574c) #8 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x36226) #9 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x425fa) #10 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x5625f) #11 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x5625f) #12 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x41ee2) #13 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x351b) #14 main /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2cc6) Location is heap block of size 1984 at 0x7b70000d0000 allocated by main thread: #0 operator new(unsigned long) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:57 (ts_tablet_manager-itest+0x4da083) #1 kudu::master::MiniMaster::Start() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/master/mini_master.cc:92:29 (libmaster.so+0x1a3237) #2 kudu::master::MiniMaster::Restart() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/master/mini_master.cc:110:10 (libmaster.so+0x1a3640) #3 kudu::tserver::TsTabletManagerITest_TestTableStats_Test::TestBody() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/ts_tablet_manager-itest.cc:737:7 (ts_tablet_manager-itest+0x4e4a77) #4 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x552ef) #5 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x552ef) #6 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474:5 (libgmock.so+0x344b8) #7 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x3574c) #8 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x36226) #9 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x425fa) #10 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x5625f) #11 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x5625f) #12 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x41ee2) #13 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x351b) #14 main /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2cc6) Mutex M35153 (0x7b70000d0430) created at: #0 pthread_mutex_init /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1211 (ts_tablet_manager-itest+0x45cf25) #1 kudu::Mutex::Mutex() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/mutex.cc:65:8 (libkudu_util.so+0x1b5f6f) #2 kudu::CountDownLatch::CountDownLatch(int) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/countdown_latch.h:34:12 (libitest_util.so+0x7ed3d) #3 kudu::Promise::Promise() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/promise.h:33:15 (libmaster.so+0x18563e) #4 kudu::master::Master::Master(kudu::master::MasterOptions const&) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/master/master.cc:112:9 (libmaster.so+0x182844) #5 kudu::master::MiniMaster::Start() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/master/mini_master.cc:92:33 (libmaster.so+0x1a3249) #6 kudu::master::MiniMaster::Restart() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/master/mini_master.cc:110:10 (libmaster.so+0x1a3640) #7 kudu::tserver::TsTabletManagerITest_TestTableStats_Test::TestBody() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/ts_tablet_manager-itest.cc:737:7 (ts_tablet_manager-itest+0x4e4a77) #8 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x552ef) #9 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x552ef) #10 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474:5 (libgmock.so+0x344b8) #11 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x3574c) #12 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x36226) #13 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x425fa) #14 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x5625f) #15 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x5625f) #16 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x41ee2) #17 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x351b) #18 main /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2cc6) Thread T264 'init [worker]-1' (tid=1608, finished) created by main thread at: #0 pthread_create /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992 (ts_tablet_manager-itest+0x44aa76) #1 kudu::Thread::StartThread(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, boost::function const&, unsigned long, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.cc:601:15 (libkudu_util.so+0x1ed95b) #2 kudu::Status kudu::Thread::Create(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, void (kudu::ThreadPool::* const&)(), kudu::ThreadPool* const&, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.h:164:12 (libkudu_util.so+0x1fb995) #3 kudu::ThreadPool::CreateThread() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:749:10 (libkudu_util.so+0x1f8c82) #4 kudu::ThreadPool::DoSubmit(std::__1::shared_ptr, kudu::ThreadPoolToken*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:556:21 (libkudu_util.so+0x1f744f) #5 kudu::ThreadPool::Submit(std::__1::shared_ptr) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:458:10 (libkudu_util.so+0x1f8eef) #6 kudu::ThreadPool::SubmitClosure(kudu::Callback) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:450:10 (libkudu_util.so+0x1f8e31) #7 kudu::master::Master::StartAsync() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/master/master.cc:195:3 (libmaster.so+0x183870) #8 kudu::master::MiniMaster::Start() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/master/mini_master.cc:94:3 (libmaster.so+0x1a3284) #9 kudu::master::MiniMaster::Restart() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/master/mini_master.cc:110:10 (libmaster.so+0x1a3640) #10 kudu::tserver::TsTabletManagerITest_TestTableStats_Test::TestBody() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/ts_tablet_manager-itest.cc:737:7 (ts_tablet_manager-itest+0x4e4a77) #11 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x552ef) #12 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x552ef) #13 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474:5 (libgmock.so+0x344b8) #14 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x3574c) #15 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x36226) #16 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x425fa) #17 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x5625f) #18 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x5625f) #19 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x41ee2) #20 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x351b) #21 main /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2cc6) SUMMARY: ThreadSanitizer: data race /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/condition_variable.cc:46:12 in kudu::ConditionVariable::~ConditionVariable() ================== ================== WARNING: ThreadSanitizer: data race (pid=244) Write of size 1 at 0x7b70000d0430 by main thread: #0 pthread_mutex_destroy /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1228 (ts_tablet_manager-itest+0x453bff) #1 kudu::Mutex::~Mutex() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/mutex.cc:76:12 (libkudu_util.so+0x1b61c0) #2 kudu::CountDownLatch::~CountDownLatch() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/countdown_latch.h:31:7 (libitest_util.so+0x80792) #3 kudu::Promise::~Promise() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/promise.h:34:15 (libmaster.so+0x185845) #4 kudu::master::Master::~Master() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/master/master.cc:128:1 (libmaster.so+0x182c42) #5 kudu::master::Master::~Master() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/master/master.cc:126:19 (libmaster.so+0x182e99) #6 std::__1::default_delete::operator()(kudu::master::Master*) const /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/c++/v1/memory:2285:5 (libmaster.so+0x1a3093) #7 std::__1::unique_ptr >::reset(kudu::master::Master*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/c++/v1/memory:2598 (libmaster.so+0x1a3093) #8 kudu::master::MiniMaster::Shutdown() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/master/mini_master.cc:118 (libmaster.so+0x1a3093) #9 kudu::tserver::TsTabletManagerITest_TestTableStats_Test::TestBody() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/ts_tablet_manager-itest.cc:735:20 (ts_tablet_manager-itest+0x4e4a4d) #10 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x552ef) #11 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x552ef) #12 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474:5 (libgmock.so+0x344b8) #13 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x3574c) #14 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x36226) #15 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x425fa) #16 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x5625f) #17 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x5625f) #18 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x41ee2) #19 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x351b) #20 main /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2cc6) Previous atomic read of size 1 at 0x7b70000d0430 by thread T264: #0 pthread_mutex_trylock /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1237 (ts_tablet_manager-itest+0x4571c2) #1 kudu::Mutex::TryAcquire() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/mutex.cc:81:12 (libkudu_util.so+0x1b62b0) #2 kudu::Mutex::Acquire() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/mutex.cc:98:7 (libkudu_util.so+0x1b6672) #3 kudu::MutexLock::MutexLock(kudu::Mutex&) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/mutex.h:94:12 (libitest_util.so+0x80145) #4 kudu::CountDownLatch::CountDown(int) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/countdown_latch.h:44:15 (libitest_util.so+0x7f28f) #5 kudu::CountDownLatch::CountDown() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/countdown_latch.h:65:5 (libitest_util.so+0x7f36e) #6 kudu::Promise::Set(kudu::Status const&) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/promise.h:69:12 (libmaster.so+0x185dcb) #7 kudu::master::Master::InitCatalogManagerTask() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/master/master.cc:207:16 (libmaster.so+0x183f52) #8 kudu::internal::RunnableAdapter::Run(kudu::master::Master*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/bind_internal.h:136:12 (libmaster.so+0x188489) #9 kudu::internal::InvokeHelper, void (kudu::master::Master*)>::MakeItSo(kudu::internal::RunnableAdapter, kudu::master::Master*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/bind_internal.h:873:14 (libmaster.so+0x1883a5) #10 kudu::internal::Invoker<1, kudu::internal::BindState, void (kudu::master::Master*), void (kudu::internal::UnretainedWrapper)>, void (kudu::master::Master*)>::Run(kudu::internal::BindStateBase*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/bind_internal.h:1065:12 (libmaster.so+0x1882ea) #11 kudu::Callback::Run() const /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/callback.h:396:12 (libconsensus.so+0xa97fd) #12 kudu::ClosureRunnable::Run() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:76:9 (libkudu_util.so+0x1fd94d) #13 kudu::ThreadPool::DispatchThread() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:686:22 (libkudu_util.so+0x1f9678) #14 boost::_mfi::mf0::operator()(kudu::ThreadPool*) const /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkudu_util.so+0x2045e9) #15 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkudu_util.so+0x20453a) #16 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkudu_util.so+0x2044c3) #17 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkudu_util.so+0x2042b9) #18 boost::function0::operator()() const /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xba0b1) #19 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.cc:657:3 (libkudu_util.so+0x1ee174) As if synchronized via sleep: #0 nanosleep /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:355 (ts_tablet_manager-itest+0x450b0a) #1 base::SleepForNanoseconds(long) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/sysinfo.cc:95:10 (libgutil.so+0x61d02) #2 kudu::SleepFor(kudu::MonoDelta const&) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/monotime.cc:267:3 (libkudu_util.so+0x1b5826) #3 kudu::tserver::TsTabletManagerITest_TestTableStats_Test::TestBody() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/ts_tablet_manager-itest.cc:741:7 (ts_tablet_manager-itest+0x4e4b0a) #4 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x552ef) #5 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x552ef) #6 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474:5 (libgmock.so+0x344b8) #7 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x3574c) #8 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x36226) #9 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x425fa) #10 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x5625f) #11 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x5625f) #12 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x41ee2) #13 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x351b) #14 main /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2cc6) Location is heap block of size 1984 at 0x7b70000d0000 allocated by main thread: #0 operator new(unsigned long) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:57 (ts_tablet_manager-itest+0x4da083) #1 kudu::master::MiniMaster::Start() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/master/mini_master.cc:92:29 (libmaster.so+0x1a3237) #2 kudu::master::MiniMaster::Restart() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/master/mini_master.cc:110:10 (libmaster.so+0x1a3640) #3 kudu::tserver::TsTabletManagerITest_TestTableStats_Test::TestBody() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/ts_tablet_manager-itest.cc:737:7 (ts_tablet_manager-itest+0x4e4a77) #4 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x552ef) #5 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x552ef) #6 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474:5 (libgmock.so+0x344b8) #7 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x3574c) #8 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x36226) #9 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x425fa) #10 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x5625f) #11 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x5625f) #12 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x41ee2) #13 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x351b) #14 main /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2cc6) Thread T264 'init [worker]-1' (tid=1608, finished) created by main thread at: #0 pthread_create /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992 (ts_tablet_manager-itest+0x44aa76) #1 kudu::Thread::StartThread(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, boost::function const&, unsigned long, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.cc:601:15 (libkudu_util.so+0x1ed95b) #2 kudu::Status kudu::Thread::Create(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, void (kudu::ThreadPool::* const&)(), kudu::ThreadPool* const&, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.h:164:12 (libkudu_util.so+0x1fb995) #3 kudu::ThreadPool::CreateThread() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:749:10 (libkudu_util.so+0x1f8c82) #4 kudu::ThreadPool::DoSubmit(std::__1::shared_ptr, kudu::ThreadPoolToken*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:556:21 (libkudu_util.so+0x1f744f) #5 kudu::ThreadPool::Submit(std::__1::shared_ptr) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:458:10 (libkudu_util.so+0x1f8eef) #6 kudu::ThreadPool::SubmitClosure(kudu::Callback) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:450:10 (libkudu_util.so+0x1f8e31) #7 kudu::master::Master::StartAsync() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/master/master.cc:195:3 (libmaster.so+0x183870) #8 kudu::master::MiniMaster::Start() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/master/mini_master.cc:94:3 (libmaster.so+0x1a3284) #9 kudu::master::MiniMaster::Restart() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/master/mini_master.cc:110:10 (libmaster.so+0x1a3640) #10 kudu::tserver::TsTabletManagerITest_TestTableStats_Test::TestBody() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/ts_tablet_manager-itest.cc:737:7 (ts_tablet_manager-itest+0x4e4a77) #11 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x552ef) #12 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x552ef) #13 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474:5 (libgmock.so+0x344b8) #14 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x3574c) #15 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x36226) #16 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x425fa) #17 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x5625f) #18 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x5625f) #19 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x41ee2) #20 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x351b) #21 main /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2cc6) SUMMARY: ThreadSanitizer: data race /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/mutex.cc:76:12 in kudu::Mutex::~Mutex() ================== I0916 07:55:44.390360 1547 raft_consensus.cc:472] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5fe1f662b7a54c0f823ca2d656131416) I0916 07:55:44.390848 1547 raft_consensus.cc:494] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } I0916 07:55:44.392216 1547 leader_election.cc:296] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5fe1f662b7a54c0f823ca2d656131416 (127.0.61.61:37305), d933e518c06d4c7d8b4f743167c2c643 (127.0.61.60:37785) W0916 07:55:44.393469 1020 leader_election.cc:341] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 5fe1f662b7a54c0f823ca2d656131416 (127.0.61.61:37305): Network error: Client connection negotiation failed: client connection to 127.0.61.61:37305: connect: Connection refused (error 111) I0916 07:55:44.402231 1661 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 17 } } ignore_live_leader: false dest_uuid: "d933e518c06d4c7d8b4f743167c2c643" is_pre_election: true I0916 07:55:44.402813 1661 raft_consensus.cc:2311] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fd1e33dc408e40c7a8bae2850d5a6b6f for term 4 because replica is either leader or believes a valid leader to be alive. I0916 07:55:44.403652 1019 leader_election.cc:310] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd1e33dc408e40c7a8bae2850d5a6b6f; no voters: 5fe1f662b7a54c0f823ca2d656131416, d933e518c06d4c7d8b4f743167c2c643 I0916 07:55:44.404314 1547 raft_consensus.cc:2592] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority I0916 07:55:44.428954 1688 raft_consensus.cc:472] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5fe1f662b7a54c0f823ca2d656131416) I0916 07:55:44.429332 1688 raft_consensus.cc:494] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } I0916 07:55:44.430946 1688 leader_election.cc:296] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fd1e33dc408e40c7a8bae2850d5a6b6f (127.0.61.62:33603), 5fe1f662b7a54c0f823ca2d656131416 (127.0.61.61:37305) W0916 07:55:44.439663 1634 leader_election.cc:341] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 5fe1f662b7a54c0f823ca2d656131416 (127.0.61.61:37305): Network error: Client connection negotiation failed: client connection to 127.0.61.61:37305: connect: Connection refused (error 111) I0916 07:55:44.441987 1047 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d933e518c06d4c7d8b4f743167c2c643" candidate_term: 4 candidate_status { last_received { term: 3 index: 17 } } ignore_live_leader: false dest_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" is_pre_election: true I0916 07:55:44.442500 1047 raft_consensus.cc:2350] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d933e518c06d4c7d8b4f743167c2c643 in term 3. I0916 07:55:44.443356 1635 leader_election.cc:310] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d933e518c06d4c7d8b4f743167c2c643, fd1e33dc408e40c7a8bae2850d5a6b6f; no voters: 5fe1f662b7a54c0f823ca2d656131416 I0916 07:55:44.444034 1688 raft_consensus.cc:2644] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 3 FOLLOWER]: Leader pre-election won for term 4 I0916 07:55:44.444279 1688 raft_consensus.cc:472] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5fe1f662b7a54c0f823ca2d656131416) I0916 07:55:44.444555 1688 raft_consensus.cc:2890] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 3 FOLLOWER]: Advancing to term 4 I0916 07:55:44.448112 1688 raft_consensus.cc:494] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } I0916 07:55:44.449271 1688 leader_election.cc:296] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [CANDIDATE]: Term 4 election: Requested vote from peers fd1e33dc408e40c7a8bae2850d5a6b6f (127.0.61.62:33603), 5fe1f662b7a54c0f823ca2d656131416 (127.0.61.61:37305) I0916 07:55:44.450040 1047 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d933e518c06d4c7d8b4f743167c2c643" candidate_term: 4 candidate_status { last_received { term: 3 index: 17 } } ignore_live_leader: false dest_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" W0916 07:55:44.450366 1634 leader_election.cc:341] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 5fe1f662b7a54c0f823ca2d656131416 (127.0.61.61:37305): Network error: Client connection negotiation failed: client connection to 127.0.61.61:37305: connect: Connection refused (error 111) I0916 07:55:44.450582 1047 raft_consensus.cc:2890] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 3 FOLLOWER]: Advancing to term 4 I0916 07:55:44.454154 1047 raft_consensus.cc:2350] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d933e518c06d4c7d8b4f743167c2c643 in term 4. I0916 07:55:44.454886 1635 leader_election.cc:310] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d933e518c06d4c7d8b4f743167c2c643, fd1e33dc408e40c7a8bae2850d5a6b6f; no voters: 5fe1f662b7a54c0f823ca2d656131416 I0916 07:55:44.455471 1688 raft_consensus.cc:2644] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 4 FOLLOWER]: Leader election won for term 4 I0916 07:55:44.455771 1688 raft_consensus.cc:667] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 4 LEADER]: Becoming Leader. State: Replica: d933e518c06d4c7d8b4f743167c2c643, State: Running, Role: LEADER I0916 07:55:44.456176 1688 consensus_queue.cc:206] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 3.17, Last appended by leader: 17, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } I0916 07:55:44.457914 1693 sys_catalog.cc:340] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d933e518c06d4c7d8b4f743167c2c643. Latest consensus state: current_term: 4 leader_uuid: "d933e518c06d4c7d8b4f743167c2c643" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:44.458395 1693 sys_catalog.cc:343] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [sys.catalog]: This master's current role is: LEADER I0916 07:55:44.459679 1696 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0916 07:55:44.462280 1696 catalog_manager.cc:387] Loaded metadata for table NewTableName [id=074536fca6144499906b7c13e035c370] I0916 07:55:44.464978 1696 catalog_manager.cc:441] Loaded metadata for tablet 6b5196040b094455ad661e9ab9e379c7 (table NewTableName [id=074536fca6144499906b7c13e035c370]) I0916 07:55:44.465571 1696 catalog_manager.cc:441] Loaded metadata for tablet e02eeaf40c1d4a299b70f4a8f1514e31 (table NewTableName [id=074536fca6144499906b7c13e035c370]) I0916 07:55:44.466159 1696 catalog_manager.cc:441] Loaded metadata for tablet e5cce06208f44e4fa384db94c0b624ae (table NewTableName [id=074536fca6144499906b7c13e035c370]) I0916 07:55:44.466634 1696 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0916 07:55:44.469008 1696 catalog_manager.cc:1106] Loading token signing keys... I0916 07:55:44.470985 1696 catalog_manager.cc:4446] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643: Loaded TSK: 0 I0916 07:55:44.490600 244 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 48016us I0916 07:55:44.490993 244 fs_manager.cc:263] Metadata directory not provided I0916 07:55:44.491137 244 fs_manager.cc:266] Using existing metadata directory in first data directory I0916 07:55:44.493486 244 fs_manager.cc:399] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s I0916 07:55:44.494577 244 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0916 07:55:44.505023 244 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0916 07:55:44.505230 244 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/master-1-root uuid: "5fe1f662b7a54c0f823ca2d656131416" format_stamp: "Formatted at 2019-09-16 07:55:39 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:44.505460 244 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/master-1-root metadata directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/master-1-root 1 data directories: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/master-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0916 07:55:44.543551 1634 consensus_peers.cc:458] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 -> Peer 5fe1f662b7a54c0f823ca2d656131416 (127.0.61.61:37305): Couldn't send request to peer 5fe1f662b7a54c0f823ca2d656131416. Status: Network error: Client connection negotiation failed: client connection to 127.0.61.61:37305: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0916 07:55:44.555801 1047 raft_consensus.cc:1184] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 4 FOLLOWER]: Refusing update from remote peer d933e518c06d4c7d8b4f743167c2c643: Log matching property violated. Preceding OpId in replica: term: 3 index: 17. Preceding OpId from leader: term: 4 index: 18. (index mismatch) I0916 07:55:44.557050 1693 consensus_queue.cc:984] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s I0916 07:55:44.560060 1547 sys_catalog.cc:340] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d933e518c06d4c7d8b4f743167c2c643. Latest consensus state: current_term: 4 leader_uuid: "d933e518c06d4c7d8b4f743167c2c643" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:44.561760 1547 sys_catalog.cc:343] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [sys.catalog]: This master's current role is: FOLLOWER I0916 07:55:44.564159 1693 sys_catalog.cc:340] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "d933e518c06d4c7d8b4f743167c2c643" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:44.564923 1693 sys_catalog.cc:343] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [sys.catalog]: This master's current role is: LEADER I0916 07:55:44.566520 1547 sys_catalog.cc:340] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "d933e518c06d4c7d8b4f743167c2c643" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:44.567359 1547 sys_catalog.cc:343] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [sys.catalog]: This master's current role is: FOLLOWER I0916 07:55:44.581683 244 rpc_server.cc:205] RPC server started. Bound to: 127.0.61.61:37305 I0916 07:55:44.583813 244 webserver.cc:374] Webserver started at http://127.0.61.61:38341/ using document root and password file I0916 07:55:44.589586 1751 sys_catalog.cc:188] Verifying existing consensus state I0916 07:55:44.593117 1751 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416: Bootstrap starting. I0916 07:55:44.611762 1415 heartbeater.cc:447] Master 127.0.61.60:37785 was elected leader, sending a full tablet report... I0916 07:55:44.619626 1344 heartbeater.cc:447] Master 127.0.61.60:37785 was elected leader, sending a full tablet report... I0916 07:55:44.623188 1273 heartbeater.cc:447] Master 127.0.61.60:37785 was elected leader, sending a full tablet report... I0916 07:55:44.625075 1274 heartbeater.cc:322] Connected to a master server at 127.0.61.61:37305 I0916 07:55:44.631744 1345 heartbeater.cc:322] Connected to a master server at 127.0.61.61:37305 I0916 07:55:44.632236 1416 heartbeater.cc:322] Connected to a master server at 127.0.61.61:37305 I0916 07:55:44.641309 1751 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416: Bootstrap replayed 1/1 log segments. Stats: ops{read=17 overwritten=0 applied=17 ignored=0} inserts{seen=6 ignored=0} mutations{seen=13 ignored=0} orphaned_commits=0. Pending: 0 replicates I0916 07:55:44.642208 1751 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416: Bootstrap complete. I0916 07:55:44.644328 1751 raft_consensus.cc:340] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } I0916 07:55:44.645025 1751 raft_consensus.cc:705] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fe1f662b7a54c0f823ca2d656131416, State: Initialized, Role: FOLLOWER I0916 07:55:44.645525 1751 consensus_queue.cc:229] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 17, Last appended: 3.17, Last appended by leader: 17, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } I0916 07:55:44.647138 1756 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:44.647699 1756 sys_catalog.cc:343] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: This master's current role is: FOLLOWER I0916 07:55:44.647020 1751 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:44.648067 1751 sys_catalog.cc:343] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: This master's current role is: FOLLOWER I0916 07:55:44.648906 1751 sys_catalog.cc:428] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: configured and running, proceeding with master startup. I0916 07:55:44.651101 1728 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 3 FOLLOWER]: Advancing to term 4 I0916 07:55:44.655234 1757 catalog_manager.cc:1142] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416: acquiring CA information for follower catalog manager: success I0916 07:55:44.658628 1757 catalog_manager.cc:1170] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0916 07:55:44.660769 1760 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "d933e518c06d4c7d8b4f743167c2c643" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:44.661039 1756 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d933e518c06d4c7d8b4f743167c2c643. Latest consensus state: current_term: 4 leader_uuid: "d933e518c06d4c7d8b4f743167c2c643" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:44.661564 1760 sys_catalog.cc:343] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: This master's current role is: FOLLOWER I0916 07:55:44.661708 1756 sys_catalog.cc:343] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [sys.catalog]: This master's current role is: FOLLOWER I0916 07:55:44.687566 244 tablet_server.cc:159] TabletServer@127.0.61.1:40619 shutting down... I0916 07:55:44.697623 1716 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" instance_seqno: 1568620540262017) as {username='slave'} at 127.0.0.1:42898; Asking this server to re-register. I0916 07:55:44.699116 1416 heartbeater.cc:406] Registering TS with master... I0916 07:55:44.700026 1718 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" instance_seqno: 1568620539861819) as {username='slave'} at 127.0.0.1:42900; Asking this server to re-register. I0916 07:55:44.693023 1717 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" instance_seqno: 1568620540049147) as {username='slave'} at 127.0.0.1:42902; Asking this server to re-register. I0916 07:55:44.702361 1274 heartbeater.cc:406] Registering TS with master... I0916 07:55:44.702500 1345 heartbeater.cc:406] Registering TS with master... I0916 07:55:44.704617 1718 ts_manager.cc:161] Registered new tserver with Master: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:44.705060 1717 ts_manager.cc:161] Registered new tserver with Master: bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401) I0916 07:55:44.705737 1716 ts_manager.cc:161] Registered new tserver with Master: 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619) I0916 07:55:44.723377 244 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0916 07:55:44.723748 244 tablet_replica.cc:273] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709: stopping tablet replica I0916 07:55:44.724056 244 raft_consensus.cc:2147] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 3 LEADER]: Raft consensus shutting down. I0916 07:55:44.724658 244 raft_consensus.cc:2174] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 3 FOLLOWER]: Raft consensus is shut down! I0916 07:55:44.726783 244 tablet_replica.cc:273] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709: stopping tablet replica I0916 07:55:44.727063 244 raft_consensus.cc:2147] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 2 LEADER]: Raft consensus shutting down. I0916 07:55:44.727560 244 raft_consensus.cc:2174] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 2 FOLLOWER]: Raft consensus is shut down! I0916 07:55:44.729475 244 tablet_replica.cc:273] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709: stopping tablet replica I0916 07:55:44.729744 244 raft_consensus.cc:2147] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 3 FOLLOWER]: Raft consensus shutting down. I0916 07:55:44.729988 244 raft_consensus.cc:2174] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 3 FOLLOWER]: Raft consensus is shut down! I0916 07:55:44.751215 244 tablet_server.cc:173] TabletServer@127.0.61.1:40619 shutdown complete. I0916 07:55:44.758954 1693 sys_catalog.cc:340] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "d933e518c06d4c7d8b4f743167c2c643" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd1e33dc408e40c7a8bae2850d5a6b6f" member_type: VOTER last_known_addr { host: "127.0.61.62" port: 33603 } } peers { permanent_uuid: "5fe1f662b7a54c0f823ca2d656131416" member_type: VOTER last_known_addr { host: "127.0.61.61" port: 37305 } } peers { permanent_uuid: "d933e518c06d4c7d8b4f743167c2c643" member_type: VOTER last_known_addr { host: "127.0.61.60" port: 37785 } } } I0916 07:55:44.760008 1693 sys_catalog.cc:343] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [sys.catalog]: This master's current role is: LEADER W0916 07:55:44.771878 1283 consensus_peers.cc:458] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f -> Peer 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619): Couldn't send request to peer 39273ba79fdb4d31abbb34278b8b1709. Status: Network error: Client connection negotiation failed: client connection to 127.0.61.1:40619: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0916 07:55:44.776546 244 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 48016us I0916 07:55:44.776895 244 fs_manager.cc:263] Metadata directory not provided I0916 07:55:44.777004 244 fs_manager.cc:266] Using existing metadata directory in first data directory I0916 07:55:44.779363 244 fs_manager.cc:399] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s I0916 07:55:44.780670 244 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0916 07:55:44.788658 244 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0916 07:55:44.788838 244 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-0-root uuid: "39273ba79fdb4d31abbb34278b8b1709" format_stamp: "Formatted at 2019-09-16 07:55:39 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:44.789052 244 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0916 07:55:44.812520 244 ts_tablet_manager.cc:371] Loading tablet metadata (0/3 complete) I0916 07:55:44.822866 244 ts_tablet_manager.cc:383] Loaded tablet metadata (3 total tablets, 3 live tablets) I0916 07:55:44.823057 244 ts_tablet_manager.cc:389] Registering tablets (0/3 complete) I0916 07:55:44.827116 1772 tablet_bootstrap.cc:439] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709: Bootstrap starting. I0916 07:55:44.834766 1774 raft_consensus.cc:472] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 39273ba79fdb4d31abbb34278b8b1709) I0916 07:55:44.835162 1774 raft_consensus.cc:494] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:44.837337 1320 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bce8dbcd72a840bb8256dee0715b175f" is_pre_election: true W0916 07:55:44.838127 1354 leader_election.cc:341] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619): Network error: Client connection negotiation failed: client connection to 127.0.61.1:40619: connect: Connection refused (error 111) I0916 07:55:44.838161 1774 leader_election.cc:296] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401), 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619) I0916 07:55:44.838160 1320 raft_consensus.cc:2311] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 for term 4 because replica is either leader or believes a valid leader to be alive. I0916 07:55:44.839607 1353 leader_election.cc:310] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90; no voters: 39273ba79fdb4d31abbb34278b8b1709, bce8dbcd72a840bb8256dee0715b175f I0916 07:55:44.840318 1774 raft_consensus.cc:2592] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority I0916 07:55:44.840728 1618 raft_consensus.cc:472] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 39273ba79fdb4d31abbb34278b8b1709) I0916 07:55:44.841153 1618 raft_consensus.cc:494] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:44.840423 244 ts_tablet_manager.cc:403] Registered 3 tablets I0916 07:55:44.842878 1618 leader_election.cc:296] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619), 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) W0916 07:55:44.844039 1283 leader_election.cc:341] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619): Network error: Client connection negotiation failed: client connection to 127.0.61.1:40619: connect: Connection refused (error 111) I0916 07:55:44.844799 1392 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "bce8dbcd72a840bb8256dee0715b175f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" is_pre_election: true I0916 07:55:44.845397 1392 raft_consensus.cc:2350] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bce8dbcd72a840bb8256dee0715b175f in term 3. I0916 07:55:44.846588 1285 leader_election.cc:310] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, bce8dbcd72a840bb8256dee0715b175f; no voters: 39273ba79fdb4d31abbb34278b8b1709 I0916 07:55:44.847615 1618 raft_consensus.cc:2644] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 3 FOLLOWER]: Leader pre-election won for term 4 I0916 07:55:44.847903 1618 raft_consensus.cc:472] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 39273ba79fdb4d31abbb34278b8b1709) I0916 07:55:44.848166 1618 raft_consensus.cc:2890] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 3 FOLLOWER]: Advancing to term 4 I0916 07:55:44.852330 1618 raft_consensus.cc:494] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:44.854321 1618 leader_election.cc:296] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 4 election: Requested vote from peers 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619), 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:44.854465 1392 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "bce8dbcd72a840bb8256dee0715b175f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" I0916 07:55:44.855119 1392 raft_consensus.cc:2890] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 3 FOLLOWER]: Advancing to term 4 W0916 07:55:44.855166 1283 leader_election.cc:341] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619): Network error: Client connection negotiation failed: client connection to 127.0.61.1:40619: connect: Connection refused (error 111) I0916 07:55:44.858747 1392 raft_consensus.cc:2350] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bce8dbcd72a840bb8256dee0715b175f in term 4. I0916 07:55:44.859552 1285 leader_election.cc:310] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, bce8dbcd72a840bb8256dee0715b175f; no voters: 39273ba79fdb4d31abbb34278b8b1709 I0916 07:55:44.860167 1618 raft_consensus.cc:2644] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 4 FOLLOWER]: Leader election won for term 4 I0916 07:55:44.871409 1618 raft_consensus.cc:667] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 4 LEADER]: Becoming Leader. State: Replica: bce8dbcd72a840bb8256dee0715b175f, State: Running, Role: LEADER I0916 07:55:44.873014 1777 raft_consensus.cc:472] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39273ba79fdb4d31abbb34278b8b1709) I0916 07:55:44.876497 1777 raft_consensus.cc:494] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:44.875717 1618 consensus_queue.cc:206] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } W0916 07:55:44.881659 1283 leader_election.cc:341] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619): Network error: Client connection negotiation failed: client connection to 127.0.61.1:40619: connect: Connection refused (error 111) I0916 07:55:44.881954 1392 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e5cce06208f44e4fa384db94c0b624ae" candidate_uuid: "bce8dbcd72a840bb8256dee0715b175f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" is_pre_election: true I0916 07:55:44.882730 1392 raft_consensus.cc:2311] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bce8dbcd72a840bb8256dee0715b175f for term 3 because replica is either leader or believes a valid leader to be alive. I0916 07:55:44.884011 1285 leader_election.cc:310] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bce8dbcd72a840bb8256dee0715b175f; no voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, 39273ba79fdb4d31abbb34278b8b1709 I0916 07:55:44.880933 1777 leader_election.cc:296] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619), 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:44.885586 1777 raft_consensus.cc:2592] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0916 07:55:44.884104 1772 tablet.cc:1223] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709: Alter schema from ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0916 07:55:44.887303 1772 tablet_metadata.cc:569] Not flushing: waiting for 1 pins to be released. I0916 07:55:44.888320 1772 tablet_bootstrap.cc:439] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=99 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0916 07:55:44.888384 1650 catalog_manager.cc:4120] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f reported cstate change: term changed from 3 to 4, leader changed from 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1) to bce8dbcd72a840bb8256dee0715b175f (127.0.61.2). New cstate: current_term: 4 leader_uuid: "bce8dbcd72a840bb8256dee0715b175f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } health_report { overall_health: UNKNOWN } } } I0916 07:55:44.889761 1772 tablet_bootstrap.cc:439] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709: Bootstrap complete. I0916 07:55:44.895223 1774 raft_consensus.cc:472] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39273ba79fdb4d31abbb34278b8b1709) I0916 07:55:44.895656 1774 raft_consensus.cc:494] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:44.897388 1774 leader_election.cc:296] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619), bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401) I0916 07:55:44.898444 1320 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e5cce06208f44e4fa384db94c0b624ae" candidate_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "bce8dbcd72a840bb8256dee0715b175f" is_pre_election: true I0916 07:55:44.899045 1320 raft_consensus.cc:2350] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 in term 2. W0916 07:55:44.899487 1354 leader_election.cc:341] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619): Network error: Client connection negotiation failed: client connection to 127.0.61.1:40619: connect: Connection refused (error 111) I0916 07:55:44.899890 1353 leader_election.cc:310] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, bce8dbcd72a840bb8256dee0715b175f; no voters: 39273ba79fdb4d31abbb34278b8b1709 I0916 07:55:44.900874 1774 raft_consensus.cc:2644] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 2 FOLLOWER]: Leader pre-election won for term 3 I0916 07:55:44.901144 1774 raft_consensus.cc:472] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 39273ba79fdb4d31abbb34278b8b1709) I0916 07:55:44.901389 1774 raft_consensus.cc:2890] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 2 FOLLOWER]: Advancing to term 3 I0916 07:55:44.909014 1774 raft_consensus.cc:494] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:44.940356 1320 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e5cce06208f44e4fa384db94c0b624ae" candidate_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "bce8dbcd72a840bb8256dee0715b175f" W0916 07:55:44.941001 1354 leader_election.cc:341] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619): Network error: Client connection negotiation failed: client connection to 127.0.61.1:40619: connect: Connection refused (error 111) I0916 07:55:44.941308 1320 raft_consensus.cc:2890] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 2 FOLLOWER]: Advancing to term 3 I0916 07:55:44.942353 1774 leader_election.cc:296] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 3 election: Requested vote from peers 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619), bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401) I0916 07:55:44.953727 1320 raft_consensus.cc:2350] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 in term 3. I0916 07:55:44.955072 1353 leader_election.cc:310] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, bce8dbcd72a840bb8256dee0715b175f; no voters: 39273ba79fdb4d31abbb34278b8b1709 I0916 07:55:44.955797 1772 ts_tablet_manager.cc:1085] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709: Time spent bootstrapping tablet: real 0.129s user 0.050s sys 0.030s I0916 07:55:44.956498 1774 raft_consensus.cc:2644] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 3 FOLLOWER]: Leader election won for term 3 I0916 07:55:44.958693 1774 raft_consensus.cc:667] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 3 LEADER]: Becoming Leader. State: Replica: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, State: Running, Role: LEADER I0916 07:55:44.958604 1772 raft_consensus.cc:340] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:44.959491 1772 raft_consensus.cc:705] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39273ba79fdb4d31abbb34278b8b1709, State: Initialized, Role: FOLLOWER I0916 07:55:44.959668 1774 consensus_queue.cc:206] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:44.960589 1772 consensus_queue.cc:229] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:44.963932 1772 ts_tablet_manager.cc:1113] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s I0916 07:55:44.964826 1772 tablet_bootstrap.cc:439] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709: Bootstrap starting. I0916 07:55:44.967571 1650 catalog_manager.cc:4120] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 reported cstate change: term changed from 2 to 3, leader changed from 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1) to 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3). New cstate: current_term: 3 leader_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } health_report { overall_health: HEALTHY } } } I0916 07:55:44.991710 1392 raft_consensus.cc:1184] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 4 FOLLOWER]: Refusing update from remote peer bce8dbcd72a840bb8256dee0715b175f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch) I0916 07:55:44.993214 1618 consensus_queue.cc:984] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s W0916 07:55:44.998301 1283 consensus_peers.cc:458] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f -> Peer 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619): Couldn't send request to peer 39273ba79fdb4d31abbb34278b8b1709. Status: Network error: Client connection negotiation failed: client connection to 127.0.61.1:40619: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0916 07:55:45.045915 1772 tablet.cc:1223] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709: Alter schema from ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0916 07:55:45.046624 1772 tablet_metadata.cc:569] Not flushing: waiting for 1 pins to be released. I0916 07:55:45.047545 1772 tablet_bootstrap.cc:439] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=111 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0916 07:55:45.049284 1772 tablet_bootstrap.cc:439] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709: Bootstrap complete. I0916 07:55:45.056039 1772 ts_tablet_manager.cc:1085] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709: Time spent bootstrapping tablet: real 0.091s user 0.057s sys 0.004s I0916 07:55:45.057502 1772 raft_consensus.cc:340] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:45.058032 1772 raft_consensus.cc:705] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39273ba79fdb4d31abbb34278b8b1709, State: Initialized, Role: FOLLOWER I0916 07:55:45.058533 1772 consensus_queue.cc:229] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:45.059850 1772 ts_tablet_manager.cc:1113] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s I0916 07:55:45.060281 1772 tablet_bootstrap.cc:439] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709: Bootstrap starting. I0916 07:55:45.065042 244 rpc_server.cc:205] RPC server started. Bound to: 127.0.61.1:40619 I0916 07:55:45.066886 244 webserver.cc:374] Webserver started at http://127.0.61.1:34949/ using document root and password file I0916 07:55:45.072171 1320 raft_consensus.cc:1184] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 3 FOLLOWER]: Refusing update from remote peer 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch) I0916 07:55:45.074640 1794 consensus_queue.cc:984] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0916 07:55:45.153923 1798 raft_consensus.cc:472] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0916 07:55:45.154570 1798 raft_consensus.cc:494] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:45.156525 1798 leader_election.cc:296] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401), 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:45.192124 1822 raft_consensus.cc:2890] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 2 FOLLOWER]: Advancing to term 3 I0916 07:55:45.198385 1772 tablet.cc:1223] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709: Alter schema from ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0916 07:55:45.199010 1772 tablet_metadata.cc:569] Not flushing: waiting for 1 pins to be released. I0916 07:55:45.199805 1772 tablet_bootstrap.cc:439] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=97 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0916 07:55:45.200655 1772 tablet_bootstrap.cc:439] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709: Bootstrap complete. I0916 07:55:45.201303 1822 raft_consensus.cc:1184] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 3 FOLLOWER]: Refusing update from remote peer 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch) I0916 07:55:45.206648 1847 heartbeater.cc:322] Connected to a master server at 127.0.61.60:37785 I0916 07:55:45.207016 1847 heartbeater.cc:406] Registering TS with master... I0916 07:55:45.207160 1774 consensus_queue.cc:984] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0916 07:55:45.208672 1847 heartbeater.cc:455] Master 127.0.61.60:37785 requested a full tablet report, sending... I0916 07:55:45.212081 1821 raft_consensus.cc:2890] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 3 FOLLOWER]: Advancing to term 4 I0916 07:55:45.239661 1848 heartbeater.cc:322] Connected to a master server at 127.0.61.61:37305 I0916 07:55:45.240023 1848 heartbeater.cc:406] Registering TS with master... I0916 07:55:45.240612 1848 heartbeater.cc:455] Master 127.0.61.61:37305 requested a full tablet report, sending... I0916 07:55:45.242048 1650 ts_manager.cc:161] Re-registered known tserver with Master: 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619) I0916 07:55:45.246263 1650 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41200 I0916 07:55:45.246172 1320 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e5cce06208f44e4fa384db94c0b624ae" candidate_uuid: "39273ba79fdb4d31abbb34278b8b1709" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "bce8dbcd72a840bb8256dee0715b175f" is_pre_election: true I0916 07:55:45.246834 1320 raft_consensus.cc:2311] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39273ba79fdb4d31abbb34278b8b1709 for term 3 because replica is either leader or believes a valid leader to be alive. I0916 07:55:45.249330 1847 heartbeater.cc:447] Master 127.0.61.60:37785 was elected leader, sending a full tablet report... I0916 07:55:45.252372 1772 ts_tablet_manager.cc:1085] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709: Time spent bootstrapping tablet: real 0.192s user 0.035s sys 0.012s I0916 07:55:45.254036 1772 raft_consensus.cc:340] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } I0916 07:55:45.254539 1772 raft_consensus.cc:705] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39273ba79fdb4d31abbb34278b8b1709, State: Initialized, Role: FOLLOWER I0916 07:55:45.255200 1772 consensus_queue.cc:229] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } I0916 07:55:45.257246 1772 ts_tablet_manager.cc:1113] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s I0916 07:55:45.263857 1716 ts_manager.cc:161] Re-registered known tserver with Master: 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619) I0916 07:55:45.264137 1392 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e5cce06208f44e4fa384db94c0b624ae" candidate_uuid: "39273ba79fdb4d31abbb34278b8b1709" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" is_pre_election: true I0916 07:55:45.265653 1392 raft_consensus.cc:2311] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate 39273ba79fdb4d31abbb34278b8b1709 for term 3 because replica is either leader or believes a valid leader to be alive. I0916 07:55:45.266925 1767 leader_election.cc:310] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39273ba79fdb4d31abbb34278b8b1709; no voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, bce8dbcd72a840bb8256dee0715b175f I0916 07:55:45.268395 1798 raft_consensus.cc:2592] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0916 07:55:45.274782 1853 heartbeater.cc:322] Connected to a master server at 127.0.61.62:33603 I0916 07:55:45.275146 1853 heartbeater.cc:406] Registering TS with master... I0916 07:55:45.275566 1853 heartbeater.cc:455] Master 127.0.61.62:33603 requested a full tablet report, sending... I0916 07:55:45.278420 1036 ts_manager.cc:161] Re-registered known tserver with Master: 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619) I0916 07:55:45.360491 244 tablet_server.cc:159] TabletServer@127.0.61.2:39401 shutting down... I0916 07:55:45.401767 244 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0916 07:55:45.402133 244 tablet_replica.cc:273] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f: stopping tablet replica I0916 07:55:45.402556 244 raft_consensus.cc:2147] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 4 LEADER]: Raft consensus shutting down. I0916 07:55:45.403064 244 raft_consensus.cc:2174] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 4 FOLLOWER]: Raft consensus is shut down! I0916 07:55:45.403177 1282 messenger.cc:381] Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer W0916 07:55:45.403873 1353 consensus_peers.cc:458] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 -> Peer bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401): Couldn't send request to peer bce8dbcd72a840bb8256dee0715b175f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry. I0916 07:55:45.405755 244 tablet_replica.cc:273] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f: stopping tablet replica I0916 07:55:45.406075 244 raft_consensus.cc:2147] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 3 FOLLOWER]: Raft consensus shutting down. I0916 07:55:45.406291 244 raft_consensus.cc:2174] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 3 FOLLOWER]: Raft consensus is shut down! I0916 07:55:45.408641 244 tablet_replica.cc:273] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f: stopping tablet replica I0916 07:55:45.408926 244 raft_consensus.cc:2147] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 3 LEADER]: Raft consensus shutting down. I0916 07:55:45.409430 244 raft_consensus.cc:2174] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 3 FOLLOWER]: Raft consensus is shut down! I0916 07:55:45.431604 244 tablet_server.cc:173] TabletServer@127.0.61.2:39401 shutdown complete. I0916 07:55:45.456874 244 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 48516us I0916 07:55:45.457217 244 fs_manager.cc:263] Metadata directory not provided I0916 07:55:45.457327 244 fs_manager.cc:266] Using existing metadata directory in first data directory I0916 07:55:45.459524 244 fs_manager.cc:399] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s I0916 07:55:45.460748 244 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0916 07:55:45.468073 244 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0916 07:55:45.468240 244 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-1-root uuid: "bce8dbcd72a840bb8256dee0715b175f" format_stamp: "Formatted at 2019-09-16 07:55:39 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:45.468452 244 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0916 07:55:45.503799 244 ts_tablet_manager.cc:383] Loaded tablet metadata (3 total tablets, 3 live tablets) I0916 07:55:45.507949 1876 tablet_bootstrap.cc:439] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f: Bootstrap starting. I0916 07:55:45.515806 244 ts_tablet_manager.cc:403] Registered 3 tablets I0916 07:55:45.541818 1876 tablet.cc:1223] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f: Alter schema from ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0916 07:55:45.542385 1876 tablet_metadata.cc:569] Not flushing: waiting for 1 pins to be released. I0916 07:55:45.543682 1876 tablet_bootstrap.cc:439] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=99 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0916 07:55:45.544333 1876 tablet_bootstrap.cc:439] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f: Bootstrap complete. I0916 07:55:45.544797 1865 raft_consensus.cc:472] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bce8dbcd72a840bb8256dee0715b175f) I0916 07:55:45.545400 1865 raft_consensus.cc:494] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } I0916 07:55:45.548051 1821 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6b5196040b094455ad661e9ab9e379c7" candidate_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "39273ba79fdb4d31abbb34278b8b1709" is_pre_election: true I0916 07:55:45.548738 1821 raft_consensus.cc:2311] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 for term 4 because replica is either leader or believes a valid leader to be alive. I0916 07:55:45.550963 1865 leader_election.cc:296] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619), bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401) W0916 07:55:45.552016 1353 leader_election.cc:341] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401): Network error: Client connection negotiation failed: client connection to 127.0.61.2:39401: connect: Connection refused (error 111) I0916 07:55:45.552408 1353 leader_election.cc:310] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90; no voters: 39273ba79fdb4d31abbb34278b8b1709, bce8dbcd72a840bb8256dee0715b175f I0916 07:55:45.553251 1865 raft_consensus.cc:2592] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority I0916 07:55:45.556594 1876 ts_tablet_manager.cc:1085] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f: Time spent bootstrapping tablet: real 0.049s user 0.034s sys 0.004s I0916 07:55:45.558842 1876 raft_consensus.cc:340] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 3 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:45.559630 1876 raft_consensus.cc:705] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: bce8dbcd72a840bb8256dee0715b175f, State: Initialized, Role: FOLLOWER I0916 07:55:45.561055 1876 consensus_queue.cc:229] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:45.564232 1876 ts_tablet_manager.cc:1113] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s I0916 07:55:45.564888 1876 tablet_bootstrap.cc:439] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f: Bootstrap starting. I0916 07:55:45.579541 1798 raft_consensus.cc:472] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bce8dbcd72a840bb8256dee0715b175f) I0916 07:55:45.580111 1798 raft_consensus.cc:494] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } I0916 07:55:45.581995 1798 leader_election.cc:296] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451), bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401) W0916 07:55:45.583830 1764 leader_election.cc:341] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401): Network error: Client connection negotiation failed: client connection to 127.0.61.2:39401: connect: Connection refused (error 111) I0916 07:55:45.585461 1392 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6b5196040b094455ad661e9ab9e379c7" candidate_uuid: "39273ba79fdb4d31abbb34278b8b1709" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" is_pre_election: true I0916 07:55:45.586119 1392 raft_consensus.cc:2350] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39273ba79fdb4d31abbb34278b8b1709 in term 3. I0916 07:55:45.586217 1865 raft_consensus.cc:472] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader bce8dbcd72a840bb8256dee0715b175f) I0916 07:55:45.586766 1865 raft_consensus.cc:494] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:45.587358 1767 leader_election.cc:310] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, 39273ba79fdb4d31abbb34278b8b1709; no voters: bce8dbcd72a840bb8256dee0715b175f I0916 07:55:45.588510 1798 raft_consensus.cc:2644] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 3 FOLLOWER]: Leader pre-election won for term 4 I0916 07:55:45.588840 1798 raft_consensus.cc:472] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 3 FOLLOWER]: Starting leader election (detected failure of leader bce8dbcd72a840bb8256dee0715b175f) I0916 07:55:45.589123 1798 raft_consensus.cc:2890] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 3 FOLLOWER]: Advancing to term 4 I0916 07:55:45.591147 1821 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39273ba79fdb4d31abbb34278b8b1709" is_pre_election: true I0916 07:55:45.591934 1821 raft_consensus.cc:2311] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 for term 5 because replica is either leader or believes a valid leader to be alive. W0916 07:55:45.593189 1353 leader_election.cc:341] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401): Network error: Client connection negotiation failed: client connection to 127.0.61.2:39401: connect: Connection refused (error 111) I0916 07:55:45.593555 1353 leader_election.cc:310] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90; no voters: 39273ba79fdb4d31abbb34278b8b1709, bce8dbcd72a840bb8256dee0715b175f I0916 07:55:45.594938 1794 raft_consensus.cc:2592] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I0916 07:55:45.588446 1865 leader_election.cc:296] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401), 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619) I0916 07:55:45.596969 1798 raft_consensus.cc:494] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } I0916 07:55:45.599251 1392 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6b5196040b094455ad661e9ab9e379c7" candidate_uuid: "39273ba79fdb4d31abbb34278b8b1709" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" I0916 07:55:45.599764 1392 raft_consensus.cc:2890] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 3 FOLLOWER]: Advancing to term 4 I0916 07:55:45.608273 1392 raft_consensus.cc:2350] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39273ba79fdb4d31abbb34278b8b1709 in term 4. I0916 07:55:45.611203 1767 leader_election.cc:310] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, 39273ba79fdb4d31abbb34278b8b1709; no voters: I0916 07:55:45.614986 1876 tablet.cc:1223] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f: Alter schema from ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0916 07:55:45.615804 1876 tablet_metadata.cc:569] Not flushing: waiting for 1 pins to be released. I0916 07:55:45.626315 1798 leader_election.cc:296] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 4 election: Requested vote from peers 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451), bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401) I0916 07:55:45.626925 1916 raft_consensus.cc:472] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader bce8dbcd72a840bb8256dee0715b175f) I0916 07:55:45.627072 1915 raft_consensus.cc:2644] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 4 FOLLOWER]: Leader election won for term 4 I0916 07:55:45.627480 1915 raft_consensus.cc:667] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 4 LEADER]: Becoming Leader. State: Replica: 39273ba79fdb4d31abbb34278b8b1709, State: Running, Role: LEADER I0916 07:55:45.627421 1916 raft_consensus.cc:494] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:45.628069 1915 consensus_queue.cc:206] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } I0916 07:55:45.629021 1916 leader_election.cc:296] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401), 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:45.629950 1876 tablet_bootstrap.cc:439] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=111 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0916 07:55:45.630048 1392 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "39273ba79fdb4d31abbb34278b8b1709" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" is_pre_election: true I0916 07:55:45.630774 1392 raft_consensus.cc:2350] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39273ba79fdb4d31abbb34278b8b1709 in term 4. I0916 07:55:45.631623 1767 leader_election.cc:310] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, 39273ba79fdb4d31abbb34278b8b1709; no voters: I0916 07:55:45.632664 1916 raft_consensus.cc:2644] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 4 FOLLOWER]: Leader pre-election won for term 5 I0916 07:55:45.632979 1916 raft_consensus.cc:472] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 4 FOLLOWER]: Starting leader election (detected failure of leader bce8dbcd72a840bb8256dee0715b175f) I0916 07:55:45.633291 1916 raft_consensus.cc:2890] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 4 FOLLOWER]: Advancing to term 5 I0916 07:55:45.632735 1650 catalog_manager.cc:4120] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 reported cstate change: term changed from 3 to 4, leader changed from bce8dbcd72a840bb8256dee0715b175f (127.0.61.2) to 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1). New cstate: current_term: 4 leader_uuid: "39273ba79fdb4d31abbb34278b8b1709" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } health_report { overall_health: UNKNOWN } } } I0916 07:55:45.630826 1876 tablet_bootstrap.cc:439] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f: Bootstrap complete. W0916 07:55:45.639163 1764 leader_election.cc:341] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401): Network error: Client connection negotiation failed: client connection to 127.0.61.2:39401: connect: Connection refused (error 111) I0916 07:55:45.638984 1916 raft_consensus.cc:494] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } W0916 07:55:45.639861 1764 leader_election.cc:341] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401): Network error: Client connection negotiation failed: client connection to 127.0.61.2:39401: connect: Connection refused (error 111) I0916 07:55:45.641386 1916 leader_election.cc:296] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 5 election: Requested vote from peers bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401), 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:45.643376 1392 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "39273ba79fdb4d31abbb34278b8b1709" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" I0916 07:55:45.644037 1392 raft_consensus.cc:2890] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 4 FOLLOWER]: Advancing to term 5 I0916 07:55:45.650094 1392 raft_consensus.cc:2350] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39273ba79fdb4d31abbb34278b8b1709 in term 5. I0916 07:55:45.651157 1767 leader_election.cc:310] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, 39273ba79fdb4d31abbb34278b8b1709; no voters: I0916 07:55:45.652222 1916 raft_consensus.cc:2644] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 5 FOLLOWER]: Leader election won for term 5 I0916 07:55:45.652871 1916 raft_consensus.cc:667] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 5 LEADER]: Becoming Leader. State: Replica: 39273ba79fdb4d31abbb34278b8b1709, State: Running, Role: LEADER I0916 07:55:45.654316 1916 consensus_queue.cc:206] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } W0916 07:55:45.655614 1764 leader_election.cc:341] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401): Network error: Client connection negotiation failed: client connection to 127.0.61.2:39401: connect: Connection refused (error 111) I0916 07:55:45.687417 1876 ts_tablet_manager.cc:1085] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f: Time spent bootstrapping tablet: real 0.123s user 0.047s sys 0.011s I0916 07:55:45.693974 1876 raft_consensus.cc:340] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 4 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:45.695173 1876 raft_consensus.cc:705] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: bce8dbcd72a840bb8256dee0715b175f, State: Initialized, Role: FOLLOWER I0916 07:55:45.696710 1876 consensus_queue.cc:229] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:45.700179 1876 ts_tablet_manager.cc:1113] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s I0916 07:55:45.702472 1876 tablet_bootstrap.cc:439] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f: Bootstrap starting. I0916 07:55:45.702620 1650 catalog_manager.cc:4120] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 reported cstate change: term changed from 4 to 5, leader changed from bce8dbcd72a840bb8256dee0715b175f (127.0.61.2) to 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1). New cstate: current_term: 5 leader_uuid: "39273ba79fdb4d31abbb34278b8b1709" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } health_report { overall_health: UNKNOWN } } } I0916 07:55:45.720793 244 rpc_server.cc:205] RPC server started. Bound to: 127.0.61.2:39401 I0916 07:55:45.723456 244 webserver.cc:374] Webserver started at http://127.0.61.2:37889/ using document root and password file I0916 07:55:45.751823 1392 raft_consensus.cc:1184] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 4 FOLLOWER]: Refusing update from remote peer 39273ba79fdb4d31abbb34278b8b1709: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch) I0916 07:55:45.754431 1916 consensus_queue.cc:984] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s I0916 07:55:45.789270 1391 raft_consensus.cc:1184] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 5 FOLLOWER]: Refusing update from remote peer 39273ba79fdb4d31abbb34278b8b1709: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch) I0916 07:55:45.791435 1915 consensus_queue.cc:984] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0916 07:55:45.859016 1902 raft_consensus.cc:472] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0916 07:55:45.860618 1902 raft_consensus.cc:494] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:45.862298 1902 leader_election.cc:296] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619), 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:45.862957 1902 raft_consensus.cc:472] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0916 07:55:45.863283 1902 raft_consensus.cc:494] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:45.864840 1902 leader_election.cc:296] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619), 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:45.906464 1949 heartbeater.cc:322] Connected to a master server at 127.0.61.61:37305 I0916 07:55:45.906849 1949 heartbeater.cc:406] Registering TS with master... I0916 07:55:45.908632 1949 heartbeater.cc:455] Master 127.0.61.61:37305 requested a full tablet report, sending... I0916 07:55:45.914345 1912 raft_consensus.cc:2890] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 4 FOLLOWER]: Advancing to term 5 W0916 07:55:45.914981 1764 consensus_peers.cc:458] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 -> Peer bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401): Couldn't send request to peer bce8dbcd72a840bb8256dee0715b175f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry. I0916 07:55:45.915586 1716 ts_manager.cc:161] Re-registered known tserver with Master: bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401) I0916 07:55:45.920400 1912 raft_consensus.cc:1184] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 5 FOLLOWER]: Refusing update from remote peer 39273ba79fdb4d31abbb34278b8b1709: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch) I0916 07:55:45.921576 1950 heartbeater.cc:322] Connected to a master server at 127.0.61.62:33603 I0916 07:55:45.921878 1950 heartbeater.cc:406] Registering TS with master... I0916 07:55:45.922112 1948 heartbeater.cc:322] Connected to a master server at 127.0.61.60:37785 I0916 07:55:45.921967 1915 consensus_queue.cc:984] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0916 07:55:45.922436 1948 heartbeater.cc:406] Registering TS with master... I0916 07:55:45.922435 1950 heartbeater.cc:455] Master 127.0.61.62:33603 requested a full tablet report, sending... I0916 07:55:45.927819 1036 ts_manager.cc:161] Re-registered known tserver with Master: bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401) I0916 07:55:45.929066 1948 heartbeater.cc:455] Master 127.0.61.60:37785 requested a full tablet report, sending... I0916 07:55:45.932396 1876 tablet.cc:1223] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f: Alter schema from ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0916 07:55:45.933593 1876 tablet_metadata.cc:569] Not flushing: waiting for 1 pins to be released. I0916 07:55:45.934757 1876 tablet_bootstrap.cc:439] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=97 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0916 07:55:45.935617 1876 tablet_bootstrap.cc:439] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f: Bootstrap complete. I0916 07:55:45.940209 1650 ts_manager.cc:161] Re-registered known tserver with Master: bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401) I0916 07:55:45.947958 1650 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41230 I0916 07:55:45.950598 1822 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "bce8dbcd72a840bb8256dee0715b175f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39273ba79fdb4d31abbb34278b8b1709" is_pre_election: true I0916 07:55:45.951247 1822 raft_consensus.cc:2311] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate bce8dbcd72a840bb8256dee0715b175f for term 5 because replica is either leader or believes a valid leader to be alive. I0916 07:55:45.951908 1948 heartbeater.cc:447] Master 127.0.61.60:37785 was elected leader, sending a full tablet report... I0916 07:55:45.951997 1821 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e5cce06208f44e4fa384db94c0b624ae" candidate_uuid: "bce8dbcd72a840bb8256dee0715b175f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "39273ba79fdb4d31abbb34278b8b1709" is_pre_election: true I0916 07:55:45.952754 1821 raft_consensus.cc:2311] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bce8dbcd72a840bb8256dee0715b175f for term 4 because replica is either leader or believes a valid leader to be alive. I0916 07:55:45.957078 1391 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e5cce06208f44e4fa384db94c0b624ae" candidate_uuid: "bce8dbcd72a840bb8256dee0715b175f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" is_pre_election: true I0916 07:55:45.957957 1391 raft_consensus.cc:2311] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate bce8dbcd72a840bb8256dee0715b175f for term 4 because replica is either leader or believes a valid leader to be alive. I0916 07:55:45.958817 1871 leader_election.cc:310] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bce8dbcd72a840bb8256dee0715b175f; no voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, 39273ba79fdb4d31abbb34278b8b1709 I0916 07:55:45.958784 1392 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e02eeaf40c1d4a299b70f4a8f1514e31" candidate_uuid: "bce8dbcd72a840bb8256dee0715b175f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" is_pre_election: true I0916 07:55:45.959669 1392 raft_consensus.cc:2311] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bce8dbcd72a840bb8256dee0715b175f for term 5 because replica is either leader or believes a valid leader to be alive. I0916 07:55:45.959735 1963 raft_consensus.cc:2592] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority I0916 07:55:45.960566 1876 ts_tablet_manager.cc:1085] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f: Time spent bootstrapping tablet: real 0.258s user 0.049s sys 0.013s I0916 07:55:45.960500 1871 leader_election.cc:310] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bce8dbcd72a840bb8256dee0715b175f; no voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, 39273ba79fdb4d31abbb34278b8b1709 I0916 07:55:45.961199 1963 raft_consensus.cc:2592] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I0916 07:55:45.962131 1876 raft_consensus.cc:340] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 3 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } I0916 07:55:45.962677 1876 raft_consensus.cc:705] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: bce8dbcd72a840bb8256dee0715b175f, State: Initialized, Role: FOLLOWER I0916 07:55:45.963148 1876 consensus_queue.cc:229] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } I0916 07:55:45.964606 1876 ts_tablet_manager.cc:1113] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s I0916 07:55:45.970633 1911 raft_consensus.cc:2890] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 3 FOLLOWER]: Advancing to term 4 I0916 07:55:46.026479 244 tablet_server.cc:159] TabletServer@127.0.61.3:45451 shutting down... I0916 07:55:46.065549 244 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0916 07:55:46.065850 244 tablet_replica.cc:273] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: stopping tablet replica I0916 07:55:46.066238 244 raft_consensus.cc:2147] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 5 FOLLOWER]: Raft consensus shutting down. I0916 07:55:46.066555 244 raft_consensus.cc:2174] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 5 FOLLOWER]: Raft consensus is shut down! I0916 07:55:46.066565 1355 messenger.cc:381] Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer W0916 07:55:46.067390 1767 consensus_peers.cc:458] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 -> Peer 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451): Couldn't send request to peer 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry. I0916 07:55:46.069283 244 tablet_replica.cc:273] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: stopping tablet replica I0916 07:55:46.069684 244 raft_consensus.cc:2147] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 4 FOLLOWER]: Raft consensus shutting down. I0916 07:55:46.069952 244 raft_consensus.cc:2174] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 4 FOLLOWER]: Raft consensus is shut down! I0916 07:55:46.072896 244 tablet_replica.cc:273] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: stopping tablet replica I0916 07:55:46.073218 244 raft_consensus.cc:2147] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 3 LEADER]: Raft consensus shutting down. I0916 07:55:46.073710 244 raft_consensus.cc:2174] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 3 FOLLOWER]: Raft consensus is shut down! I0916 07:55:46.074190 1355 messenger.cc:381] Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer W0916 07:55:46.074998 1767 consensus_peers.cc:458] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 -> Peer 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451): Couldn't send request to peer 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry. I0916 07:55:46.089794 244 tablet_server.cc:173] TabletServer@127.0.61.3:45451 shutdown complete. I0916 07:55:46.115896 244 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 49016us I0916 07:55:46.116291 244 fs_manager.cc:263] Metadata directory not provided I0916 07:55:46.116430 244 fs_manager.cc:266] Using existing metadata directory in first data directory I0916 07:55:46.118762 244 fs_manager.cc:399] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s I0916 07:55:46.120309 244 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0916 07:55:46.128208 244 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0916 07:55:46.128389 244 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-2-root uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" format_stamp: "Formatted at 2019-09-16 07:55:40 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:46.128659 244 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0916 07:55:46.166198 244 ts_tablet_manager.cc:371] Loading tablet metadata (0/3 complete) I0916 07:55:46.178730 244 ts_tablet_manager.cc:383] Loaded tablet metadata (3 total tablets, 3 live tablets) I0916 07:55:46.179004 244 ts_tablet_manager.cc:389] Registering tablets (0/3 complete) I0916 07:55:46.183231 1979 tablet_bootstrap.cc:439] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Bootstrap starting. I0916 07:55:46.194203 244 ts_tablet_manager.cc:403] Registered 3 tablets I0916 07:55:46.197461 1963 raft_consensus.cc:472] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90) I0916 07:55:46.197886 1963 raft_consensus.cc:494] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:46.199224 1963 leader_election.cc:296] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619), 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:46.199959 1821 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e5cce06208f44e4fa384db94c0b624ae" candidate_uuid: "bce8dbcd72a840bb8256dee0715b175f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "39273ba79fdb4d31abbb34278b8b1709" is_pre_election: true I0916 07:55:46.200853 1821 raft_consensus.cc:2311] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bce8dbcd72a840bb8256dee0715b175f for term 4 because replica is either leader or believes a valid leader to be alive. W0916 07:55:46.201027 1871 leader_election.cc:341] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451): Network error: Client connection negotiation failed: client connection to 127.0.61.3:45451: connect: Connection refused (error 111) I0916 07:55:46.201953 1869 leader_election.cc:310] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bce8dbcd72a840bb8256dee0715b175f; no voters: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, 39273ba79fdb4d31abbb34278b8b1709 I0916 07:55:46.202826 1963 raft_consensus.cc:2592] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority I0916 07:55:46.227038 1979 tablet.cc:1223] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Alter schema from ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0916 07:55:46.227690 1979 tablet_metadata.cc:569] Not flushing: waiting for 1 pins to be released. I0916 07:55:46.229552 1979 tablet_bootstrap.cc:439] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=99 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0916 07:55:46.230841 1979 tablet_bootstrap.cc:439] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Bootstrap complete. I0916 07:55:46.242329 1979 ts_tablet_manager.cc:1085] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Time spent bootstrapping tablet: real 0.059s user 0.038s sys 0.008s I0916 07:55:46.245312 1979 raft_consensus.cc:340] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:46.246134 1979 raft_consensus.cc:705] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, State: Initialized, Role: FOLLOWER I0916 07:55:46.247061 1979 consensus_queue.cc:229] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:46.250342 1979 ts_tablet_manager.cc:1113] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s I0916 07:55:46.251011 1979 tablet_bootstrap.cc:439] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Bootstrap starting. I0916 07:55:46.266389 244 rpc_server.cc:205] RPC server started. Bound to: 127.0.61.3:45451 I0916 07:55:46.270081 244 webserver.cc:374] Webserver started at http://127.0.61.3:39227/ using document root and password file I0916 07:55:46.271986 1916 raft_consensus.cc:472] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90) I0916 07:55:46.272785 1916 raft_consensus.cc:494] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:46.274516 1916 leader_election.cc:296] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401), 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:46.275575 1911 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e5cce06208f44e4fa384db94c0b624ae" candidate_uuid: "39273ba79fdb4d31abbb34278b8b1709" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bce8dbcd72a840bb8256dee0715b175f" is_pre_election: true I0916 07:55:46.276564 1911 raft_consensus.cc:2350] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39273ba79fdb4d31abbb34278b8b1709 in term 3. I0916 07:55:46.297732 1764 leader_election.cc:310] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39273ba79fdb4d31abbb34278b8b1709, bce8dbcd72a840bb8256dee0715b175f; no voters: I0916 07:55:46.299227 1916 raft_consensus.cc:2644] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 3 FOLLOWER]: Leader pre-election won for term 4 I0916 07:55:46.299556 1916 raft_consensus.cc:472] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90) I0916 07:55:46.299862 1916 raft_consensus.cc:2890] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 3 FOLLOWER]: Advancing to term 4 I0916 07:55:46.305485 1916 raft_consensus.cc:494] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:46.308905 1911 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e5cce06208f44e4fa384db94c0b624ae" candidate_uuid: "39273ba79fdb4d31abbb34278b8b1709" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bce8dbcd72a840bb8256dee0715b175f" I0916 07:55:46.309669 1911 raft_consensus.cc:2890] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 3 FOLLOWER]: Advancing to term 4 I0916 07:55:46.313702 1911 raft_consensus.cc:2350] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39273ba79fdb4d31abbb34278b8b1709 in term 4. I0916 07:55:46.314633 1764 leader_election.cc:310] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39273ba79fdb4d31abbb34278b8b1709, bce8dbcd72a840bb8256dee0715b175f; no voters: I0916 07:55:46.315317 1915 raft_consensus.cc:2644] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 4 FOLLOWER]: Leader election won for term 4 I0916 07:55:46.315912 1915 raft_consensus.cc:667] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 4 LEADER]: Becoming Leader. State: Replica: 39273ba79fdb4d31abbb34278b8b1709, State: Running, Role: LEADER I0916 07:55:46.316546 1916 leader_election.cc:296] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [CANDIDATE]: Term 4 election: Requested vote from peers bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401), 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:46.317390 1915 consensus_queue.cc:206] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:46.336649 1979 tablet.cc:1223] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Alter schema from ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0916 07:55:46.337235 1979 tablet_metadata.cc:569] Not flushing: waiting for 1 pins to be released. I0916 07:55:46.338472 2012 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e5cce06208f44e4fa384db94c0b624ae" candidate_uuid: "39273ba79fdb4d31abbb34278b8b1709" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" is_pre_election: true I0916 07:55:46.338740 2011 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e5cce06208f44e4fa384db94c0b624ae" candidate_uuid: "39273ba79fdb4d31abbb34278b8b1709" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" I0916 07:55:46.339165 2012 raft_consensus.cc:2350] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39273ba79fdb4d31abbb34278b8b1709 in term 3. I0916 07:55:46.339916 2011 raft_consensus.cc:2325] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 39273ba79fdb4d31abbb34278b8b1709 for term 4 because replica is already servicing an update from a current leader or another vote. I0916 07:55:46.346694 1650 catalog_manager.cc:4120] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 reported cstate change: term changed from 3 to 4, leader changed from 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3) to 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1). New cstate: current_term: 4 leader_uuid: "39273ba79fdb4d31abbb34278b8b1709" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } health_report { overall_health: UNKNOWN } } } I0916 07:55:46.360214 1979 tablet_bootstrap.cc:439] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=111 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0916 07:55:46.364285 1979 tablet_bootstrap.cc:439] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Bootstrap complete. I0916 07:55:46.374364 2038 heartbeater.cc:322] Connected to a master server at 127.0.61.60:37785 I0916 07:55:46.374689 2038 heartbeater.cc:406] Registering TS with master... I0916 07:55:46.375211 2038 heartbeater.cc:455] Master 127.0.61.60:37785 requested a full tablet report, sending... I0916 07:55:46.375679 2039 heartbeater.cc:322] Connected to a master server at 127.0.61.61:37305 I0916 07:55:46.376006 2039 heartbeater.cc:406] Registering TS with master... I0916 07:55:46.376754 2039 heartbeater.cc:455] Master 127.0.61.61:37305 requested a full tablet report, sending... I0916 07:55:46.379516 1651 ts_manager.cc:161] Re-registered known tserver with Master: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:46.385440 1979 ts_tablet_manager.cc:1085] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Time spent bootstrapping tablet: real 0.135s user 0.038s sys 0.013s I0916 07:55:46.387048 1717 ts_manager.cc:161] Re-registered known tserver with Master: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:46.387373 1979 raft_consensus.cc:340] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:46.388276 1979 raft_consensus.cc:705] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, State: Initialized, Role: FOLLOWER I0916 07:55:46.390103 1979 consensus_queue.cc:229] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } I0916 07:55:46.391873 1979 ts_tablet_manager.cc:1113] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s I0916 07:55:46.392536 1979 tablet_bootstrap.cc:439] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Bootstrap starting. I0916 07:55:46.397578 1651 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41254 I0916 07:55:46.402196 2038 heartbeater.cc:447] Master 127.0.61.60:37785 was elected leader, sending a full tablet report... I0916 07:55:46.406209 1651 catalog_manager.cc:4215] TS 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) does not have the latest schema for tablet 6b5196040b094455ad661e9ab9e379c7 (table NewTableName [id=074536fca6144499906b7c13e035c370]). Expected version 1 got 0 I0916 07:55:46.407475 2042 heartbeater.cc:322] Connected to a master server at 127.0.61.62:33603 I0916 07:55:46.407896 2042 heartbeater.cc:406] Registering TS with master... I0916 07:55:46.408375 2042 heartbeater.cc:455] Master 127.0.61.62:33603 requested a full tablet report, sending... I0916 07:55:46.418805 1036 ts_manager.cc:161] Re-registered known tserver with Master: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) I0916 07:55:46.430157 1911 raft_consensus.cc:1184] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 4 FOLLOWER]: Refusing update from remote peer 39273ba79fdb4d31abbb34278b8b1709: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch) I0916 07:55:46.431185 1915 consensus_queue.cc:984] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I0916 07:55:46.446161 2011 raft_consensus.cc:2890] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 3 FOLLOWER]: Advancing to term 4 I0916 07:55:46.451702 2011 raft_consensus.cc:1184] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 4 FOLLOWER]: Refusing update from remote peer 39273ba79fdb4d31abbb34278b8b1709: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch) I0916 07:55:46.448066 1979 tablet.cc:1223] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Alter schema from ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0916 07:55:46.452880 1979 tablet_metadata.cc:569] Not flushing: waiting for 1 pins to be released. I0916 07:55:46.453537 1915 consensus_queue.cc:984] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I0916 07:55:46.454605 1979 tablet_bootstrap.cc:439] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=97 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0916 07:55:46.455453 1979 tablet_bootstrap.cc:439] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Bootstrap complete. I0916 07:55:46.470257 1979 ts_tablet_manager.cc:1085] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Time spent bootstrapping tablet: real 0.078s user 0.039s sys 0.016s I0916 07:55:46.472075 1979 raft_consensus.cc:340] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } I0916 07:55:46.472590 1979 raft_consensus.cc:705] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90, State: Initialized, Role: FOLLOWER I0916 07:55:46.473163 1979 consensus_queue.cc:229] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39273ba79fdb4d31abbb34278b8b1709" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 40619 } } peers { permanent_uuid: "2f1f69d5d2dc4d7f8a6ffc7bcfb07c90" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 45451 } } peers { permanent_uuid: "bce8dbcd72a840bb8256dee0715b175f" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39401 } } I0916 07:55:46.474460 1979 ts_tablet_manager.cc:1113] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s I0916 07:55:46.581753 1650 catalog_manager.cc:1936] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41264: table { table_id: "074536fca6144499906b7c13e035c370" table_name: "NewTableName" } I0916 07:55:46.594242 1650 catalog_manager.cc:4370] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643: Sending DeleteTablet for 3 replicas of tablet 6b5196040b094455ad661e9ab9e379c7 I0916 07:55:46.594888 1650 catalog_manager.cc:3512] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 6b5196040b094455ad661e9ab9e379c7 on 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619) (Table deleted at 2019-09-16 07:55:46 UTC) I0916 07:55:46.595774 1650 catalog_manager.cc:3512] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 6b5196040b094455ad661e9ab9e379c7 on 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) (Table deleted at 2019-09-16 07:55:46 UTC) I0916 07:55:46.596132 1810 tablet_service.cc:1066] Processing DeleteTablet for tablet 6b5196040b094455ad661e9ab9e379c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2019-09-16 07:55:46 UTC) from {username='slave'} at 127.0.0.1:35302 I0916 07:55:46.596503 1650 catalog_manager.cc:3512] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 6b5196040b094455ad661e9ab9e379c7 on bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401) (Table deleted at 2019-09-16 07:55:46 UTC) I0916 07:55:46.598794 1650 catalog_manager.cc:4370] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643: Sending DeleteTablet for 3 replicas of tablet e02eeaf40c1d4a299b70f4a8f1514e31 I0916 07:55:46.599182 1650 catalog_manager.cc:3512] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e02eeaf40c1d4a299b70f4a8f1514e31 on bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401) (Table deleted at 2019-09-16 07:55:46 UTC) I0916 07:55:46.599799 1650 catalog_manager.cc:3512] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e02eeaf40c1d4a299b70f4a8f1514e31 on 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619) (Table deleted at 2019-09-16 07:55:46 UTC) I0916 07:55:46.600370 1650 catalog_manager.cc:3512] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e02eeaf40c1d4a299b70f4a8f1514e31 on 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) (Table deleted at 2019-09-16 07:55:46 UTC) I0916 07:55:46.600973 1650 catalog_manager.cc:4370] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643: Sending DeleteTablet for 3 replicas of tablet e5cce06208f44e4fa384db94c0b624ae I0916 07:55:46.601164 1810 tablet_service.cc:1066] Processing DeleteTablet for tablet e02eeaf40c1d4a299b70f4a8f1514e31 with delete_type TABLET_DATA_DELETED (Table deleted at 2019-09-16 07:55:46 UTC) from {username='slave'} at 127.0.0.1:35302 I0916 07:55:46.601281 1650 catalog_manager.cc:3512] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e5cce06208f44e4fa384db94c0b624ae on 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619) (Table deleted at 2019-09-16 07:55:46 UTC) I0916 07:55:46.601824 1650 catalog_manager.cc:3512] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e5cce06208f44e4fa384db94c0b624ae on bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401) (Table deleted at 2019-09-16 07:55:46 UTC) I0916 07:55:46.602301 1810 tablet_service.cc:1066] Processing DeleteTablet for tablet e5cce06208f44e4fa384db94c0b624ae with delete_type TABLET_DATA_DELETED (Table deleted at 2019-09-16 07:55:46 UTC) from {username='slave'} at 127.0.0.1:35302 I0916 07:55:46.602474 1650 catalog_manager.cc:3512] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet e5cce06208f44e4fa384db94c0b624ae on 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451) (Table deleted at 2019-09-16 07:55:46 UTC) I0916 07:55:46.603986 2061 tablet_replica.cc:273] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709: stopping tablet replica I0916 07:55:46.604379 2061 raft_consensus.cc:2147] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 4 LEADER]: Raft consensus shutting down. I0916 07:55:46.606011 2061 raft_consensus.cc:2174] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709 [term 4 FOLLOWER]: Raft consensus is shut down! I0916 07:55:46.609892 2002 tablet_service.cc:1066] Processing DeleteTablet for tablet 6b5196040b094455ad661e9ab9e379c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2019-09-16 07:55:46 UTC) from {username='slave'} at 127.0.0.1:37622 I0916 07:55:46.610090 2061 ts_tablet_manager.cc:1440] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709: Deleting tablet data with delete state TABLET_DATA_DELETED I0916 07:55:46.610797 2001 tablet_service.cc:1066] Processing DeleteTablet for tablet e02eeaf40c1d4a299b70f4a8f1514e31 with delete_type TABLET_DATA_DELETED (Table deleted at 2019-09-16 07:55:46 UTC) from {username='slave'} at 127.0.0.1:37622 I0916 07:55:46.611361 2000 tablet_service.cc:1066] Processing DeleteTablet for tablet e5cce06208f44e4fa384db94c0b624ae with delete_type TABLET_DATA_DELETED (Table deleted at 2019-09-16 07:55:46 UTC) from {username='slave'} at 127.0.0.1:37622 I0916 07:55:46.612283 2064 tablet_replica.cc:273] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: stopping tablet replica I0916 07:55:46.616750 2064 raft_consensus.cc:2147] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 4 FOLLOWER]: Raft consensus shutting down. I0916 07:55:46.617122 2064 raft_consensus.cc:2174] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 4 FOLLOWER]: Raft consensus is shut down! I0916 07:55:46.619369 1899 tablet_service.cc:1066] Processing DeleteTablet for tablet e02eeaf40c1d4a299b70f4a8f1514e31 with delete_type TABLET_DATA_DELETED (Table deleted at 2019-09-16 07:55:46 UTC) from {username='slave'} at 127.0.0.1:40034 I0916 07:55:46.619463 1900 tablet_service.cc:1066] Processing DeleteTablet for tablet 6b5196040b094455ad661e9ab9e379c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2019-09-16 07:55:46 UTC) from {username='slave'} at 127.0.0.1:40034 I0916 07:55:46.619596 1898 tablet_service.cc:1066] Processing DeleteTablet for tablet e5cce06208f44e4fa384db94c0b624ae with delete_type TABLET_DATA_DELETED (Table deleted at 2019-09-16 07:55:46 UTC) from {username='slave'} at 127.0.0.1:40034 I0916 07:55:46.621476 2065 tablet_replica.cc:273] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f: stopping tablet replica I0916 07:55:46.622050 2065 raft_consensus.cc:2147] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 5 FOLLOWER]: Raft consensus shutting down. I0916 07:55:46.622417 2065 raft_consensus.cc:2174] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f [term 5 FOLLOWER]: Raft consensus is shut down! I0916 07:55:46.623322 2064 ts_tablet_manager.cc:1440] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Deleting tablet data with delete state TABLET_DATA_DELETED I0916 07:55:46.625779 2065 ts_tablet_manager.cc:1440] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f: Deleting tablet data with delete state TABLET_DATA_DELETED I0916 07:55:46.627020 2061 ts_tablet_manager.cc:1453] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 4.6 I0916 07:55:46.627415 2061 log.cc:990] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709: Deleting WAL directory at /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-0-root/wals/6b5196040b094455ad661e9ab9e379c7 I0916 07:55:46.628099 2061 ts_tablet_manager.cc:1474] T 6b5196040b094455ad661e9ab9e379c7 P 39273ba79fdb4d31abbb34278b8b1709: Deleting consensus metadata I0916 07:55:46.631937 1633 catalog_manager.cc:3490] TS 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619): tablet 6b5196040b094455ad661e9ab9e379c7 (table NewTableName [id=074536fca6144499906b7c13e035c370]) successfully deleted I0916 07:55:46.632817 2061 tablet_replica.cc:273] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709: stopping tablet replica I0916 07:55:46.633343 2061 raft_consensus.cc:2147] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 5 LEADER]: Raft consensus shutting down. I0916 07:55:46.634054 2061 raft_consensus.cc:2174] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709 [term 5 FOLLOWER]: Raft consensus is shut down! I0916 07:55:46.635651 2064 ts_tablet_manager.cc:1453] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 4.6 I0916 07:55:46.636070 2064 log.cc:990] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Deleting WAL directory at /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-2-root/wals/6b5196040b094455ad661e9ab9e379c7 I0916 07:55:46.636804 2064 ts_tablet_manager.cc:1474] T 6b5196040b094455ad661e9ab9e379c7 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Deleting consensus metadata I0916 07:55:46.637364 2061 ts_tablet_manager.cc:1440] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709: Deleting tablet data with delete state TABLET_DATA_DELETED I0916 07:55:46.637552 2065 ts_tablet_manager.cc:1453] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 5.7 I0916 07:55:46.637945 2065 log.cc:990] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f: Deleting WAL directory at /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-1-root/wals/e02eeaf40c1d4a299b70f4a8f1514e31 I0916 07:55:46.638630 2065 ts_tablet_manager.cc:1474] T e02eeaf40c1d4a299b70f4a8f1514e31 P bce8dbcd72a840bb8256dee0715b175f: Deleting consensus metadata I0916 07:55:46.639564 1635 catalog_manager.cc:3490] TS 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451): tablet 6b5196040b094455ad661e9ab9e379c7 (table NewTableName [id=074536fca6144499906b7c13e035c370]) successfully deleted I0916 07:55:46.640643 2064 tablet_replica.cc:273] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: stopping tablet replica I0916 07:55:46.641201 1632 catalog_manager.cc:3490] TS bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401): tablet e02eeaf40c1d4a299b70f4a8f1514e31 (table NewTableName [id=074536fca6144499906b7c13e035c370]) successfully deleted I0916 07:55:46.641289 2064 raft_consensus.cc:2147] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 5 FOLLOWER]: Raft consensus shutting down. I0916 07:55:46.641798 2064 raft_consensus.cc:2174] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 5 FOLLOWER]: Raft consensus is shut down! I0916 07:55:46.642096 2065 tablet_replica.cc:273] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f: stopping tablet replica I0916 07:55:46.644194 2065 raft_consensus.cc:2147] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 4 FOLLOWER]: Raft consensus shutting down. I0916 07:55:46.644685 2065 raft_consensus.cc:2174] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f [term 4 FOLLOWER]: Raft consensus is shut down! I0916 07:55:46.645743 2064 ts_tablet_manager.cc:1440] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Deleting tablet data with delete state TABLET_DATA_DELETED I0916 07:55:46.648180 2065 ts_tablet_manager.cc:1440] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f: Deleting tablet data with delete state TABLET_DATA_DELETED I0916 07:55:46.649518 2061 ts_tablet_manager.cc:1453] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 5.7 I0916 07:55:46.649853 2061 log.cc:990] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709: Deleting WAL directory at /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-0-root/wals/e02eeaf40c1d4a299b70f4a8f1514e31 I0916 07:55:46.650471 2061 ts_tablet_manager.cc:1474] T e02eeaf40c1d4a299b70f4a8f1514e31 P 39273ba79fdb4d31abbb34278b8b1709: Deleting consensus metadata I0916 07:55:46.653017 1633 catalog_manager.cc:3490] TS 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619): tablet e02eeaf40c1d4a299b70f4a8f1514e31 (table NewTableName [id=074536fca6144499906b7c13e035c370]) successfully deleted I0916 07:55:46.653813 2061 tablet_replica.cc:273] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709: stopping tablet replica I0916 07:55:46.654309 2061 raft_consensus.cc:2147] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 4 LEADER]: Raft consensus shutting down. I0916 07:55:46.655025 2061 raft_consensus.cc:2174] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709 [term 4 FOLLOWER]: Raft consensus is shut down! I0916 07:55:46.658109 2065 ts_tablet_manager.cc:1453] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 4.6 I0916 07:55:46.658370 2065 log.cc:990] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f: Deleting WAL directory at /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-1-root/wals/6b5196040b094455ad661e9ab9e379c7 I0916 07:55:46.658828 2065 ts_tablet_manager.cc:1474] T 6b5196040b094455ad661e9ab9e379c7 P bce8dbcd72a840bb8256dee0715b175f: Deleting consensus metadata I0916 07:55:46.660853 1632 catalog_manager.cc:3490] TS bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401): tablet 6b5196040b094455ad661e9ab9e379c7 (table NewTableName [id=074536fca6144499906b7c13e035c370]) successfully deleted I0916 07:55:46.661970 2064 ts_tablet_manager.cc:1453] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 5.7 I0916 07:55:46.662328 2064 log.cc:990] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Deleting WAL directory at /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-2-root/wals/e02eeaf40c1d4a299b70f4a8f1514e31 I0916 07:55:46.662879 2061 ts_tablet_manager.cc:1440] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709: Deleting tablet data with delete state TABLET_DATA_DELETED I0916 07:55:46.662977 2064 ts_tablet_manager.cc:1474] T e02eeaf40c1d4a299b70f4a8f1514e31 P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Deleting consensus metadata I0916 07:55:46.665844 1635 catalog_manager.cc:3490] TS 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451): tablet e02eeaf40c1d4a299b70f4a8f1514e31 (table NewTableName [id=074536fca6144499906b7c13e035c370]) successfully deleted I0916 07:55:46.666920 2065 tablet_replica.cc:273] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f: stopping tablet replica I0916 07:55:46.667003 2064 tablet_replica.cc:273] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: stopping tablet replica I0916 07:55:46.667435 2065 raft_consensus.cc:2147] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 4 FOLLOWER]: Raft consensus shutting down. I0916 07:55:46.670652 2065 raft_consensus.cc:2174] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f [term 4 FOLLOWER]: Raft consensus is shut down! I0916 07:55:46.668220 2064 raft_consensus.cc:2147] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 4 FOLLOWER]: Raft consensus shutting down. I0916 07:55:46.671196 2064 raft_consensus.cc:2174] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 [term 4 FOLLOWER]: Raft consensus is shut down! I0916 07:55:46.675431 2064 ts_tablet_manager.cc:1440] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Deleting tablet data with delete state TABLET_DATA_DELETED I0916 07:55:46.675453 2065 ts_tablet_manager.cc:1440] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f: Deleting tablet data with delete state TABLET_DATA_DELETED I0916 07:55:46.679795 2061 ts_tablet_manager.cc:1453] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 4.6 I0916 07:55:46.680141 2061 log.cc:990] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709: Deleting WAL directory at /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-0-root/wals/e5cce06208f44e4fa384db94c0b624ae I0916 07:55:46.685953 2064 ts_tablet_manager.cc:1453] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 4.6 I0916 07:55:46.686306 2064 log.cc:990] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Deleting WAL directory at /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-2-root/wals/e5cce06208f44e4fa384db94c0b624ae I0916 07:55:46.686471 2061 ts_tablet_manager.cc:1474] T e5cce06208f44e4fa384db94c0b624ae P 39273ba79fdb4d31abbb34278b8b1709: Deleting consensus metadata I0916 07:55:46.686941 2064 ts_tablet_manager.cc:1474] T e5cce06208f44e4fa384db94c0b624ae P 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90: Deleting consensus metadata I0916 07:55:46.688861 1633 catalog_manager.cc:3490] TS 39273ba79fdb4d31abbb34278b8b1709 (127.0.61.1:40619): tablet e5cce06208f44e4fa384db94c0b624ae (table NewTableName [id=074536fca6144499906b7c13e035c370]) successfully deleted I0916 07:55:46.689195 1635 catalog_manager.cc:3490] TS 2f1f69d5d2dc4d7f8a6ffc7bcfb07c90 (127.0.61.3:45451): tablet e5cce06208f44e4fa384db94c0b624ae (table NewTableName [id=074536fca6144499906b7c13e035c370]) successfully deleted I0916 07:55:46.690307 2065 ts_tablet_manager.cc:1453] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 4.6 I0916 07:55:46.690629 2065 log.cc:990] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f: Deleting WAL directory at /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/minicluster-data/ts-1-root/wals/e5cce06208f44e4fa384db94c0b624ae I0916 07:55:46.691258 2065 ts_tablet_manager.cc:1474] T e5cce06208f44e4fa384db94c0b624ae P bce8dbcd72a840bb8256dee0715b175f: Deleting consensus metadata I0916 07:55:46.693639 1632 catalog_manager.cc:3490] TS bce8dbcd72a840bb8256dee0715b175f (127.0.61.2:39401): tablet e5cce06208f44e4fa384db94c0b624ae (table NewTableName [id=074536fca6144499906b7c13e035c370]) successfully deleted I0916 07:55:46.794905 244 tablet_server.cc:159] TabletServer@127.0.61.1:40619 shutting down... I0916 07:55:46.816745 244 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0916 07:55:46.826757 244 tablet_server.cc:173] TabletServer@127.0.61.1:40619 shutdown complete. I0916 07:55:46.834888 244 tablet_server.cc:159] TabletServer@127.0.61.2:39401 shutting down... I0916 07:55:46.850920 244 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0916 07:55:46.867686 244 tablet_server.cc:173] TabletServer@127.0.61.2:39401 shutdown complete. I0916 07:55:46.877229 244 tablet_server.cc:159] TabletServer@127.0.61.3:45451 shutting down... I0916 07:55:46.893828 244 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0916 07:55:46.912185 244 tablet_server.cc:173] TabletServer@127.0.61.3:45451 shutdown complete. I0916 07:55:46.921519 244 master.cc:247] Master@127.0.61.62:33603 shutting down... I0916 07:55:46.939011 244 raft_consensus.cc:2147] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 4 FOLLOWER]: Raft consensus shutting down. I0916 07:55:46.939390 244 raft_consensus.cc:2174] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f [term 4 FOLLOWER]: Raft consensus is shut down! I0916 07:55:46.939611 244 tablet_replica.cc:273] T 00000000000000000000000000000000 P fd1e33dc408e40c7a8bae2850d5a6b6f: stopping tablet replica W0916 07:55:46.958331 244 rolling_log.cc:179] Unable to compress old log file: Not found: Unable to open input file to compress: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568620531643032-244/ts_tablet_manager-itest.dist-test-slave-dist-test-slave-p3b0.slave.diagnostics.20190916-075539.0.244: No such file or directory (error 2) I0916 07:55:46.959439 244 master.cc:258] Master@127.0.61.62:33603 shutdown complete. I0916 07:55:46.968344 244 master.cc:247] Master@127.0.61.61:37305 shutting down... I0916 07:55:46.982669 244 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 4 FOLLOWER]: Raft consensus shutting down. I0916 07:55:46.983057 244 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416 [term 4 FOLLOWER]: Raft consensus is shut down! I0916 07:55:46.983275 244 tablet_replica.cc:273] T 00000000000000000000000000000000 P 5fe1f662b7a54c0f823ca2d656131416: stopping tablet replica W0916 07:55:46.986850 1635 consensus_peers.cc:458] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 -> Peer fd1e33dc408e40c7a8bae2850d5a6b6f (127.0.61.62:33603): Couldn't send request to peer fd1e33dc408e40c7a8bae2850d5a6b6f. Status: Network error: Client connection negotiation failed: client connection to 127.0.61.62:33603: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0916 07:55:46.993965 244 master.cc:258] Master@127.0.61.61:37305 shutdown complete. I0916 07:55:47.001596 244 master.cc:247] Master@127.0.61.60:37785 shutting down... W0916 07:55:47.002647 1634 consensus_peers.cc:458] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 -> Peer 5fe1f662b7a54c0f823ca2d656131416 (127.0.61.61:37305): Couldn't send request to peer 5fe1f662b7a54c0f823ca2d656131416. Status: Network error: Client connection negotiation failed: client connection to 127.0.61.61:37305: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0916 07:55:47.018096 244 raft_consensus.cc:2147] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 4 LEADER]: Raft consensus shutting down. I0916 07:55:47.018766 244 raft_consensus.cc:2174] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643 [term 4 FOLLOWER]: Raft consensus is shut down! I0916 07:55:47.018993 244 tablet_replica.cc:273] T 00000000000000000000000000000000 P d933e518c06d4c7d8b4f743167c2c643: stopping tablet replica I0916 07:55:47.038612 244 master.cc:258] Master@127.0.61.60:37785 shutdown complete. [ OK ] TsTabletManagerITest.TestTableStats (7717 ms) [----------] 4 tests from TsTabletManagerITest (15323 ms total) [----------] 2 tests from FailedTabletsAreReplacedITest [ RUN ] FailedTabletsAreReplacedITest.OneReplica/0 I0916 07:55:47.059957 244 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.0.61.62:34307 I0916 07:55:47.071270 244 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 49516us I0916 07:55:47.071586 244 fs_manager.cc:263] Metadata directory not provided I0916 07:55:47.071702 244 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0916 07:55:47.071861 244 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0916 07:55:47.071959 244 server_base.cc:453] Attempting to create new FS layout instead I0916 07:55:47.072860 244 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568620531643032-244/minicluster-data/master-0-root/instance: uuid: "9bd8119743184be0a7f051deead359d1" format_stamp: "Formatted at 2019-09-16 07:55:47 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:47.080276 244 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s I0916 07:55:47.081593 244 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0916 07:55:47.087581 244 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0916 07:55:47.087764 244 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568620531643032-244/minicluster-data/master-0-root uuid: "9bd8119743184be0a7f051deead359d1" format_stamp: "Formatted at 2019-09-16 07:55:47 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:47.087985 244 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568620531643032-244/minicluster-data/master-0-root metadata directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568620531643032-244/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568620531643032-244/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0916 07:55:47.155539 244 rpc_server.cc:205] RPC server started. Bound to: 127.0.61.62:34307 I0916 07:55:47.157299 244 webserver.cc:374] Webserver started at http://127.0.61.62:46539/ using document root and password file I0916 07:55:47.158991 2124 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0916 07:55:47.166405 2124 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 9bd8119743184be0a7f051deead359d1: Bootstrap starting. I0916 07:55:47.168833 2124 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 9bd8119743184be0a7f051deead359d1: No blocks or log segments found. Creating new log. I0916 07:55:47.171574 2124 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 9bd8119743184be0a7f051deead359d1: No bootstrap required, opened a new log I0916 07:55:47.173260 2124 raft_consensus.cc:340] T 00000000000000000000000000000000 P 9bd8119743184be0a7f051deead359d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd8119743184be0a7f051deead359d1" member_type: VOTER } I0916 07:55:47.173596 2124 raft_consensus.cc:366] T 00000000000000000000000000000000 P 9bd8119743184be0a7f051deead359d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0916 07:55:47.173785 2124 raft_consensus.cc:705] T 00000000000000000000000000000000 P 9bd8119743184be0a7f051deead359d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bd8119743184be0a7f051deead359d1, State: Initialized, Role: FOLLOWER I0916 07:55:47.174286 2124 consensus_queue.cc:229] T 00000000000000000000000000000000 P 9bd8119743184be0a7f051deead359d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd8119743184be0a7f051deead359d1" member_type: VOTER } I0916 07:55:47.174685 2124 raft_consensus.cc:380] T 00000000000000000000000000000000 P 9bd8119743184be0a7f051deead359d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0916 07:55:47.174846 2124 raft_consensus.cc:472] T 00000000000000000000000000000000 P 9bd8119743184be0a7f051deead359d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0916 07:55:47.175041 2124 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 9bd8119743184be0a7f051deead359d1 [term 0 FOLLOWER]: Advancing to term 1 I0916 07:55:47.178911 2124 raft_consensus.cc:494] T 00000000000000000000000000000000 P 9bd8119743184be0a7f051deead359d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd8119743184be0a7f051deead359d1" member_type: VOTER } I0916 07:55:47.179355 2124 leader_election.cc:310] T 00000000000000000000000000000000 P 9bd8119743184be0a7f051deead359d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bd8119743184be0a7f051deead359d1; no voters: I0916 07:55:47.181286 2126 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 9bd8119743184be0a7f051deead359d1 [term 1 FOLLOWER]: Leader election won for term 1 I0916 07:55:47.182058 2124 leader_election.cc:296] T 00000000000000000000000000000000 P 9bd8119743184be0a7f051deead359d1 [CANDIDATE]: Term 1 election: Requested vote from peers I0916 07:55:47.183645 2126 raft_consensus.cc:667] T 00000000000000000000000000000000 P 9bd8119743184be0a7f051deead359d1 [term 1 LEADER]: Becoming Leader. State: Replica: 9bd8119743184be0a7f051deead359d1, State: Running, Role: LEADER I0916 07:55:47.184206 2126 consensus_queue.cc:206] T 00000000000000000000000000000000 P 9bd8119743184be0a7f051deead359d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd8119743184be0a7f051deead359d1" member_type: VOTER } I0916 07:55:47.186156 2124 sys_catalog.cc:340] T 00000000000000000000000000000000 P 9bd8119743184be0a7f051deead359d1 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "9bd8119743184be0a7f051deead359d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd8119743184be0a7f051deead359d1" member_type: VOTER } } I0916 07:55:47.186462 2128 sys_catalog.cc:340] T 00000000000000000000000000000000 P 9bd8119743184be0a7f051deead359d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bd8119743184be0a7f051deead359d1. Latest consensus state: current_term: 1 leader_uuid: "9bd8119743184be0a7f051deead359d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd8119743184be0a7f051deead359d1" member_type: VOTER } } I0916 07:55:47.186712 2124 sys_catalog.cc:343] T 00000000000000000000000000000000 P 9bd8119743184be0a7f051deead359d1 [sys.catalog]: This master's current role is: LEADER I0916 07:55:47.187036 2128 sys_catalog.cc:343] T 00000000000000000000000000000000 P 9bd8119743184be0a7f051deead359d1 [sys.catalog]: This master's current role is: LEADER I0916 07:55:47.186565 2127 sys_catalog.cc:340] T 00000000000000000000000000000000 P 9bd8119743184be0a7f051deead359d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bd8119743184be0a7f051deead359d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd8119743184be0a7f051deead359d1" member_type: VOTER } } I0916 07:55:47.187348 2127 sys_catalog.cc:343] T 00000000000000000000000000000000 P 9bd8119743184be0a7f051deead359d1 [sys.catalog]: This master's current role is: LEADER I0916 07:55:47.189698 2124 sys_catalog.cc:428] T 00000000000000000000000000000000 P 9bd8119743184be0a7f051deead359d1 [sys.catalog]: configured and running, proceeding with master startup. I0916 07:55:47.189877 2130 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0916 07:55:47.190666 244 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0916 07:55:47.192420 2130 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0916 07:55:47.216445 2130 catalog_manager.cc:981] Generated new certificate authority record I0916 07:55:47.218081 2130 catalog_manager.cc:1106] Loading token signing keys... I0916 07:55:47.232218 2130 catalog_manager.cc:4435] T 00000000000000000000000000000000 P 9bd8119743184be0a7f051deead359d1: Generated new TSK 0 I0916 07:55:47.271939 244 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 49516us I0916 07:55:47.272389 244 fs_manager.cc:263] Metadata directory not provided I0916 07:55:47.272632 244 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0916 07:55:47.272905 244 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0916 07:55:47.273046 244 server_base.cc:453] Attempting to create new FS layout instead I0916 07:55:47.274262 244 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568620531643032-244/minicluster-data/ts-0-root/instance: uuid: "12336a106614452ab8811f1ca5651a7f" format_stamp: "Formatted at 2019-09-16 07:55:47 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:47.282322 244 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s I0916 07:55:47.283646 244 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0916 07:55:47.289934 244 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0916 07:55:47.290098 244 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568620531643032-244/minicluster-data/ts-0-root uuid: "12336a106614452ab8811f1ca5651a7f" format_stamp: "Formatted at 2019-09-16 07:55:47 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:47.290307 244 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568620531643032-244/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568620531643032-244/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568620531643032-244/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0916 07:55:47.321538 244 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0916 07:55:47.321703 244 ts_tablet_manager.cc:403] Registered 0 tablets I0916 07:55:47.364964 244 rpc_server.cc:205] RPC server started. Bound to: 127.0.61.1:35423 I0916 07:55:47.366643 244 webserver.cc:374] Webserver started at http://127.0.61.1:43647/ using document root and password file I0916 07:55:47.381948 2198 heartbeater.cc:322] Connected to a master server at 127.0.61.62:34307 I0916 07:55:47.382293 2198 heartbeater.cc:406] Registering TS with master... I0916 07:55:47.382782 2198 heartbeater.cc:455] Master 127.0.61.62:34307 requested a full tablet report, sending... I0916 07:55:47.384613 2091 ts_manager.cc:161] Registered new tserver with Master: 12336a106614452ab8811f1ca5651a7f (127.0.61.1:35423) I0916 07:55:47.385125 244 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 49516us I0916 07:55:47.385460 244 fs_manager.cc:263] Metadata directory not provided I0916 07:55:47.385617 244 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0916 07:55:47.385805 244 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0916 07:55:47.385906 244 server_base.cc:453] Attempting to create new FS layout instead I0916 07:55:47.386389 2091 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56550 I0916 07:55:47.386823 244 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568620531643032-244/minicluster-data/ts-1-root/instance: uuid: "7c2cba5d630949b894e60e1bb10b8148" format_stamp: "Formatted at 2019-09-16 07:55:47 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:47.395172 244 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.005s sys 0.005s I0916 07:55:47.396610 244 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0916 07:55:47.403184 244 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0916 07:55:47.403362 244 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568620531643032-244/minicluster-data/ts-1-root uuid: "7c2cba5d630949b894e60e1bb10b8148" format_stamp: "Formatted at 2019-09-16 07:55:47 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:47.403569 244 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568620531643032-244/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568620531643032-244/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568620531643032-244/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0916 07:55:47.426206 244 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0916 07:55:47.426381 244 ts_tablet_manager.cc:403] Registered 0 tablets I0916 07:55:47.463095 244 rpc_server.cc:205] RPC server started. Bound to: 127.0.61.2:39907 I0916 07:55:47.464862 244 webserver.cc:374] Webserver started at http://127.0.61.2:46549/ using document root and password file I0916 07:55:47.478179 2266 heartbeater.cc:322] Connected to a master server at 127.0.61.62:34307 I0916 07:55:47.478476 2266 heartbeater.cc:406] Registering TS with master... I0916 07:55:47.478881 2266 heartbeater.cc:455] Master 127.0.61.62:34307 requested a full tablet report, sending... I0916 07:55:47.480442 2091 ts_manager.cc:161] Registered new tserver with Master: 7c2cba5d630949b894e60e1bb10b8148 (127.0.61.2:39907) I0916 07:55:47.482319 2091 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56552 I0916 07:55:47.484093 244 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 49516us I0916 07:55:47.484450 244 fs_manager.cc:263] Metadata directory not provided I0916 07:55:47.484642 244 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0916 07:55:47.484833 244 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0916 07:55:47.484935 244 server_base.cc:453] Attempting to create new FS layout instead I0916 07:55:47.485846 244 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568620531643032-244/minicluster-data/ts-2-root/instance: uuid: "dec5051aad8d4ab8927d10e1a4b937b4" format_stamp: "Formatted at 2019-09-16 07:55:47 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:47.495069 244 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.002s sys 0.008s I0916 07:55:47.496515 244 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0916 07:55:47.503888 244 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0916 07:55:47.504091 244 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568620531643032-244/minicluster-data/ts-2-root uuid: "dec5051aad8d4ab8927d10e1a4b937b4" format_stamp: "Formatted at 2019-09-16 07:55:47 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:47.504328 244 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568620531643032-244/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568620531643032-244/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568620531643032-244/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0916 07:55:47.528220 244 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0916 07:55:47.528393 244 ts_tablet_manager.cc:403] Registered 0 tablets I0916 07:55:47.572865 244 rpc_server.cc:205] RPC server started. Bound to: 127.0.61.3:38109 I0916 07:55:47.574903 244 webserver.cc:374] Webserver started at http://127.0.61.3:40611/ using document root and password file I0916 07:55:47.587168 2333 heartbeater.cc:322] Connected to a master server at 127.0.61.62:34307 I0916 07:55:47.587447 2333 heartbeater.cc:406] Registering TS with master... I0916 07:55:47.588042 2333 heartbeater.cc:455] Master 127.0.61.62:34307 requested a full tablet report, sending... I0916 07:55:47.589886 2091 ts_manager.cc:161] Registered new tserver with Master: dec5051aad8d4ab8927d10e1a4b937b4 (127.0.61.3:38109) I0916 07:55:47.590713 244 internal_mini_cluster.cc:328] 3 TS(s) registered with all masters after 0.01378606s I0916 07:55:47.591675 2091 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56554 I0916 07:55:47.609551 244 test_util.cc:212] Using random seed: -293137488 I0916 07:55:47.628789 2091 catalog_manager.cc:1440] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56558: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0916 07:55:47.665634 2165 tablet_service.cc:1022] Processing CreateTablet for tablet 0d6b8fb1ba064b57b8efeaf783085c76 (table=test-workload [id=92a032d2b20c43bd952da7b753881252]), partition=RANGE (key) PARTITION UNBOUNDED I0916 07:55:47.666539 2165 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 0d6b8fb1ba064b57b8efeaf783085c76. 1 dirs total, 0 dirs full, 0 dirs failed I0916 07:55:47.668659 2300 tablet_service.cc:1022] Processing CreateTablet for tablet 0d6b8fb1ba064b57b8efeaf783085c76 (table=test-workload [id=92a032d2b20c43bd952da7b753881252]), partition=RANGE (key) PARTITION UNBOUNDED I0916 07:55:47.669186 2233 tablet_service.cc:1022] Processing CreateTablet for tablet 0d6b8fb1ba064b57b8efeaf783085c76 (table=test-workload [id=92a032d2b20c43bd952da7b753881252]), partition=RANGE (key) PARTITION UNBOUNDED I0916 07:55:47.669646 2300 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 0d6b8fb1ba064b57b8efeaf783085c76. 1 dirs total, 0 dirs full, 0 dirs failed I0916 07:55:47.669914 2233 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 0d6b8fb1ba064b57b8efeaf783085c76. 1 dirs total, 0 dirs full, 0 dirs failed I0916 07:55:47.681401 2353 tablet_bootstrap.cc:439] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f: Bootstrap starting. I0916 07:55:47.681919 2354 tablet_bootstrap.cc:439] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4: Bootstrap starting. I0916 07:55:47.683944 2353 tablet_bootstrap.cc:590] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f: No blocks or log segments found. Creating new log. I0916 07:55:47.685683 2355 tablet_bootstrap.cc:439] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148: Bootstrap starting. I0916 07:55:47.686458 2354 tablet_bootstrap.cc:590] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4: No blocks or log segments found. Creating new log. I0916 07:55:47.688634 2353 tablet_bootstrap.cc:439] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f: No bootstrap required, opened a new log I0916 07:55:47.689046 2353 ts_tablet_manager.cc:1085] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s I0916 07:55:47.689939 2355 tablet_bootstrap.cc:590] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148: No blocks or log segments found. Creating new log. I0916 07:55:47.690132 2354 tablet_bootstrap.cc:439] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4: No bootstrap required, opened a new log I0916 07:55:47.690699 2354 ts_tablet_manager.cc:1085] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s I0916 07:55:47.690572 2353 raft_consensus.cc:340] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12336a106614452ab8811f1ca5651a7f" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 35423 } } peers { permanent_uuid: "7c2cba5d630949b894e60e1bb10b8148" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39907 } } peers { permanent_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 38109 } } I0916 07:55:47.691283 2353 raft_consensus.cc:366] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0916 07:55:47.691553 2353 raft_consensus.cc:705] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12336a106614452ab8811f1ca5651a7f, State: Initialized, Role: FOLLOWER I0916 07:55:47.692531 2353 consensus_queue.cc:229] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12336a106614452ab8811f1ca5651a7f" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 35423 } } peers { permanent_uuid: "7c2cba5d630949b894e60e1bb10b8148" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39907 } } peers { permanent_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 38109 } } I0916 07:55:47.693310 2354 raft_consensus.cc:340] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12336a106614452ab8811f1ca5651a7f" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 35423 } } peers { permanent_uuid: "7c2cba5d630949b894e60e1bb10b8148" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39907 } } peers { permanent_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 38109 } } I0916 07:55:47.694011 2354 raft_consensus.cc:366] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0916 07:55:47.694232 2354 raft_consensus.cc:705] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec5051aad8d4ab8927d10e1a4b937b4, State: Initialized, Role: FOLLOWER I0916 07:55:47.694530 2355 tablet_bootstrap.cc:439] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148: No bootstrap required, opened a new log I0916 07:55:47.694967 2355 ts_tablet_manager.cc:1085] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s I0916 07:55:47.694869 2354 consensus_queue.cc:229] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12336a106614452ab8811f1ca5651a7f" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 35423 } } peers { permanent_uuid: "7c2cba5d630949b894e60e1bb10b8148" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39907 } } peers { permanent_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 38109 } } I0916 07:55:47.697472 2198 heartbeater.cc:447] Master 127.0.61.62:34307 was elected leader, sending a full tablet report... I0916 07:55:47.697214 2355 raft_consensus.cc:340] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12336a106614452ab8811f1ca5651a7f" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 35423 } } peers { permanent_uuid: "7c2cba5d630949b894e60e1bb10b8148" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39907 } } peers { permanent_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 38109 } } I0916 07:55:47.698040 2353 ts_tablet_manager.cc:1113] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f: Time spent starting tablet: real 0.009s user 0.005s sys 0.004s I0916 07:55:47.698182 2355 raft_consensus.cc:366] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0916 07:55:47.698613 2355 raft_consensus.cc:705] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c2cba5d630949b894e60e1bb10b8148, State: Initialized, Role: FOLLOWER I0916 07:55:47.699354 2333 heartbeater.cc:447] Master 127.0.61.62:34307 was elected leader, sending a full tablet report... I0916 07:55:47.699349 2355 consensus_queue.cc:229] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12336a106614452ab8811f1ca5651a7f" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 35423 } } peers { permanent_uuid: "7c2cba5d630949b894e60e1bb10b8148" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39907 } } peers { permanent_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 38109 } } I0916 07:55:47.701122 2354 ts_tablet_manager.cc:1113] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s I0916 07:55:47.707705 2266 heartbeater.cc:447] Master 127.0.61.62:34307 was elected leader, sending a full tablet report... I0916 07:55:47.707872 2355 ts_tablet_manager.cc:1113] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s I0916 07:55:47.786119 2361 raft_consensus.cc:472] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0916 07:55:47.786592 2361 raft_consensus.cc:494] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12336a106614452ab8811f1ca5651a7f" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 35423 } } peers { permanent_uuid: "7c2cba5d630949b894e60e1bb10b8148" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39907 } } peers { permanent_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 38109 } } I0916 07:55:47.787981 2361 leader_election.cc:296] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12336a106614452ab8811f1ca5651a7f (127.0.61.1:35423), dec5051aad8d4ab8927d10e1a4b937b4 (127.0.61.3:38109) I0916 07:55:47.798801 2175 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0d6b8fb1ba064b57b8efeaf783085c76" candidate_uuid: "7c2cba5d630949b894e60e1bb10b8148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12336a106614452ab8811f1ca5651a7f" is_pre_election: true I0916 07:55:47.799347 2175 raft_consensus.cc:2350] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c2cba5d630949b894e60e1bb10b8148 in term 0. I0916 07:55:47.800248 2207 leader_election.cc:310] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12336a106614452ab8811f1ca5651a7f, 7c2cba5d630949b894e60e1bb10b8148; no voters: I0916 07:55:47.800391 2310 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0d6b8fb1ba064b57b8efeaf783085c76" candidate_uuid: "7c2cba5d630949b894e60e1bb10b8148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" is_pre_election: true I0916 07:55:47.801060 2310 raft_consensus.cc:2350] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c2cba5d630949b894e60e1bb10b8148 in term 0. I0916 07:55:47.801065 2361 raft_consensus.cc:2644] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0916 07:55:47.801467 2361 raft_consensus.cc:472] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0916 07:55:47.801676 2361 raft_consensus.cc:2890] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148 [term 0 FOLLOWER]: Advancing to term 1 I0916 07:55:47.806347 2361 raft_consensus.cc:494] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12336a106614452ab8811f1ca5651a7f" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 35423 } } peers { permanent_uuid: "7c2cba5d630949b894e60e1bb10b8148" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39907 } } peers { permanent_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 38109 } } I0916 07:55:47.807442 2361 leader_election.cc:296] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148 [CANDIDATE]: Term 1 election: Requested vote from peers 12336a106614452ab8811f1ca5651a7f (127.0.61.1:35423), dec5051aad8d4ab8927d10e1a4b937b4 (127.0.61.3:38109) I0916 07:55:47.808341 2310 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0d6b8fb1ba064b57b8efeaf783085c76" candidate_uuid: "7c2cba5d630949b894e60e1bb10b8148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" I0916 07:55:47.808275 2175 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0d6b8fb1ba064b57b8efeaf783085c76" candidate_uuid: "7c2cba5d630949b894e60e1bb10b8148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12336a106614452ab8811f1ca5651a7f" I0916 07:55:47.809010 2310 raft_consensus.cc:2890] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4 [term 0 FOLLOWER]: Advancing to term 1 I0916 07:55:47.809103 2175 raft_consensus.cc:2890] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f [term 0 FOLLOWER]: Advancing to term 1 I0916 07:55:47.814251 2310 raft_consensus.cc:2350] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c2cba5d630949b894e60e1bb10b8148 in term 1. I0916 07:55:47.814249 2175 raft_consensus.cc:2350] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c2cba5d630949b894e60e1bb10b8148 in term 1. I0916 07:55:47.815438 2204 leader_election.cc:310] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c2cba5d630949b894e60e1bb10b8148, dec5051aad8d4ab8927d10e1a4b937b4; no voters: I0916 07:55:47.816107 2361 raft_consensus.cc:2644] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148 [term 1 FOLLOWER]: Leader election won for term 1 I0916 07:55:47.817227 2361 raft_consensus.cc:667] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148 [term 1 LEADER]: Becoming Leader. State: Replica: 7c2cba5d630949b894e60e1bb10b8148, State: Running, Role: LEADER I0916 07:55:47.817710 2361 consensus_queue.cc:206] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12336a106614452ab8811f1ca5651a7f" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 35423 } } peers { permanent_uuid: "7c2cba5d630949b894e60e1bb10b8148" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39907 } } peers { permanent_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 38109 } } I0916 07:55:47.823236 2091 catalog_manager.cc:4120] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148 reported cstate change: term changed from 0 to 1, leader changed from to 7c2cba5d630949b894e60e1bb10b8148 (127.0.61.2). New cstate: current_term: 1 leader_uuid: "7c2cba5d630949b894e60e1bb10b8148" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12336a106614452ab8811f1ca5651a7f" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 35423 } } peers { permanent_uuid: "7c2cba5d630949b894e60e1bb10b8148" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39907 } } peers { permanent_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 38109 } } } Master Summary UUID | Address | Status ----------------------------------+-------------------+--------- 9bd8119743184be0a7f051deead359d1 | 127.0.61.62:34307 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+------------------+---------+---------- 12336a106614452ab8811f1ca5651a7f | 127.0.61.1:35423 | HEALTHY | 7c2cba5d630949b894e60e1bb10b8148 | 127.0.61.2:39907 | HEALTHY | dec5051aad8d4ab8927d10e1a4b937b4 | 127.0.61.3:38109 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 4 server(s) checked Tablet Summary Tablet 0d6b8fb1ba064b57b8efeaf783085c76 of table 'test-workload' is conflicted: 4 replicas' active configs disagree with the leader master's 12336a106614452ab8811f1ca5651a7f (127.0.61.1:35423): RUNNING 7c2cba5d630949b894e60e1bb10b8148 (127.0.61.2:39907): RUNNING [LEADER] dec5051aad8d4ab8927d10e1a4b937b4 (127.0.61.3:38109): RUNNING All reported replicas are: A = 12336a106614452ab8811f1ca5651a7f B = 7c2cba5d630949b894e60e1bb10b8148 C = dec5051aad8d4ab8927d10e1a4b937b4 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B C | 1 | -1 | Yes B | A B* C | 1 | -1 | Yes C | A B C | 1 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0916 07:55:48.033627 244 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0916 07:55:48.241299 2361 consensus_queue.cc:984] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12336a106614452ab8811f1ca5651a7f" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 35423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0916 07:55:48.264552 2361 consensus_queue.cc:984] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 38109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s Master Summary UUID | Address | Status ----------------------------------+-------------------+--------- 9bd8119743184be0a7f051deead359d1 | 127.0.61.62:34307 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+------------------+---------+---------- 12336a106614452ab8811f1ca5651a7f | 127.0.61.1:35423 | HEALTHY | 7c2cba5d630949b894e60e1bb10b8148 | 127.0.61.2:39907 | HEALTHY | dec5051aad8d4ab8927d10e1a4b937b4 | 127.0.61.3:38109 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 4 server(s) checked Tablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+---------+---------------+---------+------------+------------------+------------- test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0916 07:55:48.358284 244 log_verifier.cc:121] Checking tablet 0d6b8fb1ba064b57b8efeaf783085c76 I0916 07:55:48.428436 244 log_verifier.cc:172] Verified matching terms for 1 ops in tablet 0d6b8fb1ba064b57b8efeaf783085c76 I0916 07:55:48.428948 244 tablet_replica.cc:273] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148: stopping tablet replica I0916 07:55:48.429286 244 raft_consensus.cc:2147] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148 [term 1 LEADER]: Raft consensus shutting down. I0916 07:55:48.429903 244 raft_consensus.cc:2174] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148 [term 1 FOLLOWER]: Raft consensus is shut down! Master Summary UUID | Address | Status ----------------------------------+-------------------+--------- 9bd8119743184be0a7f051deead359d1 | 127.0.61.62:34307 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+------------------+---------+---------- 12336a106614452ab8811f1ca5651a7f | 127.0.61.1:35423 | HEALTHY | 7c2cba5d630949b894e60e1bb10b8148 | 127.0.61.2:39907 | HEALTHY | dec5051aad8d4ab8927d10e1a4b937b4 | 127.0.61.3:38109 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 4 server(s) checked Tablet Summary Tablet 0d6b8fb1ba064b57b8efeaf783085c76 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 12336a106614452ab8811f1ca5651a7f (127.0.61.1:35423): RUNNING 7c2cba5d630949b894e60e1bb10b8148 (127.0.61.2:39907): not running [LEADER] State: FAILED Data state: TABLET_DATA_READY Last status: IO error: INJECTED ERROR: tablet failed dec5051aad8d4ab8927d10e1a4b937b4 (127.0.61.3:38109): RUNNING All reported replicas are: A = 12336a106614452ab8811f1ca5651a7f B = 7c2cba5d630949b894e60e1bb10b8148 C = dec5051aad8d4ab8927d10e1a4b937b4 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 1 | -1 | Yes B | [config not available] | | | C | A B* C | 1 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0916 07:55:48.586791 244 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors Master Summary UUID | Address | Status ----------------------------------+-------------------+--------- 9bd8119743184be0a7f051deead359d1 | 127.0.61.62:34307 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+------------------+---------+---------- 12336a106614452ab8811f1ca5651a7f | 127.0.61.1:35423 | HEALTHY | 7c2cba5d630949b894e60e1bb10b8148 | 127.0.61.2:39907 | HEALTHY | dec5051aad8d4ab8927d10e1a4b937b4 | 127.0.61.3:38109 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 4 server(s) checked Tablet Summary Tablet 0d6b8fb1ba064b57b8efeaf783085c76 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 12336a106614452ab8811f1ca5651a7f (127.0.61.1:35423): RUNNING 7c2cba5d630949b894e60e1bb10b8148 (127.0.61.2:39907): not running [LEADER] State: FAILED Data state: TABLET_DATA_READY Last status: IO error: INJECTED ERROR: tablet failed dec5051aad8d4ab8927d10e1a4b937b4 (127.0.61.3:38109): RUNNING All reported replicas are: A = 12336a106614452ab8811f1ca5651a7f B = 7c2cba5d630949b894e60e1bb10b8148 C = dec5051aad8d4ab8927d10e1a4b937b4 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 1 | -1 | Yes B | [config not available] | | | C | A B* C | 1 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0916 07:55:48.908037 244 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors Master Summary UUID | Address | Status ----------------------------------+-------------------+--------- 9bd8119743184be0a7f051deead359d1 | 127.0.61.62:34307 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+------------------+---------+---------- 12336a106614452ab8811f1ca5651a7f | 127.0.61.1:35423 | HEALTHY | 7c2cba5d630949b894e60e1bb10b8148 | 127.0.61.2:39907 | HEALTHY | dec5051aad8d4ab8927d10e1a4b937b4 | 127.0.61.3:38109 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 4 server(s) checked Tablet Summary Tablet 0d6b8fb1ba064b57b8efeaf783085c76 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 12336a106614452ab8811f1ca5651a7f (127.0.61.1:35423): RUNNING 7c2cba5d630949b894e60e1bb10b8148 (127.0.61.2:39907): not running [LEADER] State: FAILED Data state: TABLET_DATA_READY Last status: IO error: INJECTED ERROR: tablet failed dec5051aad8d4ab8927d10e1a4b937b4 (127.0.61.3:38109): RUNNING All reported replicas are: A = 12336a106614452ab8811f1ca5651a7f B = 7c2cba5d630949b894e60e1bb10b8148 C = dec5051aad8d4ab8927d10e1a4b937b4 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 1 | -1 | Yes B | [config not available] | | | C | A B* C | 1 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0916 07:55:49.292966 244 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0916 07:55:49.757575 2508 raft_consensus.cc:472] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7c2cba5d630949b894e60e1bb10b8148) I0916 07:55:49.758059 2508 raft_consensus.cc:494] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12336a106614452ab8811f1ca5651a7f" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 35423 } } peers { permanent_uuid: "7c2cba5d630949b894e60e1bb10b8148" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39907 } } peers { permanent_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 38109 } } I0916 07:55:49.759742 2508 leader_election.cc:296] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7c2cba5d630949b894e60e1bb10b8148 (127.0.61.2:39907), dec5051aad8d4ab8927d10e1a4b937b4 (127.0.61.3:38109) I0916 07:55:49.773701 2511 raft_consensus.cc:472] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7c2cba5d630949b894e60e1bb10b8148) I0916 07:55:49.774181 2511 raft_consensus.cc:494] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12336a106614452ab8811f1ca5651a7f" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 35423 } } peers { permanent_uuid: "7c2cba5d630949b894e60e1bb10b8148" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39907 } } peers { permanent_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 38109 } } I0916 07:55:49.780572 2511 leader_election.cc:296] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12336a106614452ab8811f1ca5651a7f (127.0.61.1:35423), 7c2cba5d630949b894e60e1bb10b8148 (127.0.61.2:39907) Master Summary UUID | Address | Status ----------------------------------+-------------------+--------- 9bd8119743184be0a7f051deead359d1 | 127.0.61.62:34307 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+------------------+---------+---------- 12336a106614452ab8811f1ca5651a7f | 127.0.61.1:35423 | HEALTHY | 7c2cba5d630949b894e60e1bb10b8148 | 127.0.61.2:39907 | HEALTHY | dec5051aad8d4ab8927d10e1a4b937b4 | 127.0.61.3:38109 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 4 server(s) checked Tablet Summary Tablet 0d6b8fb1ba064b57b8efeaf783085c76 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 12336a106614452ab8811f1ca5651a7f (127.0.61.1:35423): RUNNING 7c2cba5d630949b894e60e1bb10b8148 (127.0.61.2:39907): not running [LEADER] State: FAILED Data state: TABLET_DATA_READY Last status: IO error: INJECTED ERROR: tablet failed dec5051aad8d4ab8927d10e1a4b937b4 (127.0.61.3:38109): RUNNING All reported replicas are: A = 12336a106614452ab8811f1ca5651a7f B = 7c2cba5d630949b894e60e1bb10b8148 C = dec5051aad8d4ab8927d10e1a4b937b4 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 1 | -1 | Yes B | [config not available] | | | C | A B* C | 1 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0916 07:55:49.791700 244 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0916 07:55:49.796188 2243 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0d6b8fb1ba064b57b8efeaf783085c76" candidate_uuid: "12336a106614452ab8811f1ca5651a7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7c2cba5d630949b894e60e1bb10b8148" is_pre_election: true W0916 07:55:49.797736 2136 leader_election.cc:348] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7c2cba5d630949b894e60e1bb10b8148 (127.0.61.2:39907): Illegal state: cannot vote while shut down I0916 07:55:49.799680 2310 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0d6b8fb1ba064b57b8efeaf783085c76" candidate_uuid: "12336a106614452ab8811f1ca5651a7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" is_pre_election: true I0916 07:55:49.800295 2310 raft_consensus.cc:2350] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12336a106614452ab8811f1ca5651a7f in term 1. I0916 07:55:49.801230 2136 leader_election.cc:310] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12336a106614452ab8811f1ca5651a7f, dec5051aad8d4ab8927d10e1a4b937b4; no voters: 7c2cba5d630949b894e60e1bb10b8148 I0916 07:55:49.802103 2508 raft_consensus.cc:2644] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f [term 1 FOLLOWER]: Leader pre-election won for term 2 I0916 07:55:49.802611 2508 raft_consensus.cc:472] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7c2cba5d630949b894e60e1bb10b8148) I0916 07:55:49.802906 2508 raft_consensus.cc:2890] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f [term 1 FOLLOWER]: Advancing to term 2 I0916 07:55:49.808404 2508 raft_consensus.cc:494] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12336a106614452ab8811f1ca5651a7f" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 35423 } } peers { permanent_uuid: "7c2cba5d630949b894e60e1bb10b8148" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39907 } } peers { permanent_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 38109 } } I0916 07:55:49.810552 2243 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0d6b8fb1ba064b57b8efeaf783085c76" candidate_uuid: "12336a106614452ab8811f1ca5651a7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7c2cba5d630949b894e60e1bb10b8148" I0916 07:55:49.810771 2242 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0d6b8fb1ba064b57b8efeaf783085c76" candidate_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7c2cba5d630949b894e60e1bb10b8148" is_pre_election: true W0916 07:55:49.811872 2136 leader_election.cc:348] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 7c2cba5d630949b894e60e1bb10b8148 (127.0.61.2:39907): Illegal state: cannot vote while shut down W0916 07:55:49.812018 2271 leader_election.cc:348] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7c2cba5d630949b894e60e1bb10b8148 (127.0.61.2:39907): Illegal state: cannot vote while shut down I0916 07:55:49.811962 2175 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0d6b8fb1ba064b57b8efeaf783085c76" candidate_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "12336a106614452ab8811f1ca5651a7f" is_pre_election: true I0916 07:55:49.812609 2508 leader_election.cc:296] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f [CANDIDATE]: Term 2 election: Requested vote from peers 7c2cba5d630949b894e60e1bb10b8148 (127.0.61.2:39907), dec5051aad8d4ab8927d10e1a4b937b4 (127.0.61.3:38109) I0916 07:55:49.812724 2175 raft_consensus.cc:2280] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dec5051aad8d4ab8927d10e1a4b937b4 in current term 2: Already voted for candidate 12336a106614452ab8811f1ca5651a7f in this term. I0916 07:55:49.813383 2310 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0d6b8fb1ba064b57b8efeaf783085c76" candidate_uuid: "12336a106614452ab8811f1ca5651a7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" I0916 07:55:49.813720 2274 leader_election.cc:310] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dec5051aad8d4ab8927d10e1a4b937b4; no voters: 12336a106614452ab8811f1ca5651a7f, 7c2cba5d630949b894e60e1bb10b8148 I0916 07:55:49.813796 2310 raft_consensus.cc:2890] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4 [term 1 FOLLOWER]: Advancing to term 2 I0916 07:55:49.817610 2310 raft_consensus.cc:2350] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12336a106614452ab8811f1ca5651a7f in term 2. I0916 07:55:49.817945 2511 raft_consensus.cc:2592] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0916 07:55:49.818467 2136 leader_election.cc:310] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12336a106614452ab8811f1ca5651a7f, dec5051aad8d4ab8927d10e1a4b937b4; no voters: 7c2cba5d630949b894e60e1bb10b8148 I0916 07:55:49.819224 2508 raft_consensus.cc:2644] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f [term 2 FOLLOWER]: Leader election won for term 2 I0916 07:55:49.819990 2508 raft_consensus.cc:667] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f [term 2 LEADER]: Becoming Leader. State: Replica: 12336a106614452ab8811f1ca5651a7f, State: Running, Role: LEADER I0916 07:55:49.820535 2508 consensus_queue.cc:206] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12336a106614452ab8811f1ca5651a7f" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 35423 } } peers { permanent_uuid: "7c2cba5d630949b894e60e1bb10b8148" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39907 } } peers { permanent_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 38109 } } I0916 07:55:49.824589 2090 catalog_manager.cc:4120] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f reported cstate change: term changed from 1 to 2, leader changed from 7c2cba5d630949b894e60e1bb10b8148 (127.0.61.2) to 12336a106614452ab8811f1ca5651a7f (127.0.61.1). New cstate: current_term: 2 leader_uuid: "12336a106614452ab8811f1ca5651a7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12336a106614452ab8811f1ca5651a7f" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 35423 } } peers { permanent_uuid: "7c2cba5d630949b894e60e1bb10b8148" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39907 } } peers { permanent_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 38109 } } } W0916 07:55:50.268775 2136 consensus_peers.cc:458] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f -> Peer 7c2cba5d630949b894e60e1bb10b8148 (127.0.61.2:39907): Couldn't send request to peer 7c2cba5d630949b894e60e1bb10b8148. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry. I0916 07:55:50.281930 2310 raft_consensus.cc:1184] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4 [term 2 FOLLOWER]: Refusing update from remote peer 12336a106614452ab8811f1ca5651a7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch) I0916 07:55:50.283237 2508 consensus_queue.cc:984] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 38109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s Master Summary UUID | Address | Status ----------------------------------+-------------------+--------- 9bd8119743184be0a7f051deead359d1 | 127.0.61.62:34307 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+------------------+---------+---------- 12336a106614452ab8811f1ca5651a7f | 127.0.61.1:35423 | HEALTHY | 7c2cba5d630949b894e60e1bb10b8148 | 127.0.61.2:39907 | HEALTHY | dec5051aad8d4ab8927d10e1a4b937b4 | 127.0.61.3:38109 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 4 server(s) checked Tablet Summary Tablet 0d6b8fb1ba064b57b8efeaf783085c76 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 12336a106614452ab8811f1ca5651a7f (127.0.61.1:35423): RUNNING [LEADER] 7c2cba5d630949b894e60e1bb10b8148 (127.0.61.2:39907): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: INJECTED ERROR: tablet failed dec5051aad8d4ab8927d10e1a4b937b4 (127.0.61.3:38109): RUNNING All reported replicas are: A = 12336a106614452ab8811f1ca5651a7f B = 7c2cba5d630949b894e60e1bb10b8148 C = dec5051aad8d4ab8927d10e1a4b937b4 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 2 | -1 | Yes B | [config not available] | | | C | A* B C | 2 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0916 07:55:50.465217 244 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0916 07:55:50.748255 2508 raft_consensus.cc:922] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f: Attempting to remove follower 7c2cba5d630949b894e60e1bb10b8148 from the Raft config. Reason: The tablet replica hosted on peer 7c2cba5d630949b894e60e1bb10b8148 has failed I0916 07:55:50.750232 2508 consensus_queue.cc:206] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12336a106614452ab8811f1ca5651a7f" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 35423 } } peers { permanent_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 38109 } } I0916 07:55:50.753087 2310 raft_consensus.cc:1184] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4 [term 2 FOLLOWER]: Refusing update from remote peer 12336a106614452ab8811f1ca5651a7f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch) I0916 07:55:50.754135 2515 consensus_queue.cc:984] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 38109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0916 07:55:50.758213 2508 raft_consensus.cc:2792] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 7c2cba5d630949b894e60e1bb10b8148 (127.0.61.2) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12336a106614452ab8811f1ca5651a7f" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 35423 } } peers { permanent_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 38109 } } } I0916 07:55:50.759277 2310 raft_consensus.cc:2792] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 7c2cba5d630949b894e60e1bb10b8148 (127.0.61.2) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12336a106614452ab8811f1ca5651a7f" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 35423 } } peers { permanent_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 38109 } } } I0916 07:55:50.765306 2089 catalog_manager.cc:4120] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f reported cstate change: config changed from index -1 to 3, VOTER 7c2cba5d630949b894e60e1bb10b8148 (127.0.61.2) evicted. New cstate: current_term: 2 leader_uuid: "12336a106614452ab8811f1ca5651a7f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12336a106614452ab8811f1ca5651a7f" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 35423 } } peers { permanent_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 38109 } } } I0916 07:55:50.775970 2089 catalog_manager.cc:3512] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 0d6b8fb1ba064b57b8efeaf783085c76 on 7c2cba5d630949b894e60e1bb10b8148 (127.0.61.2:39907) (TS 7c2cba5d630949b894e60e1bb10b8148 not found in new config with opid_index 3) I0916 07:55:50.776638 2089 catalog_manager.cc:3739] Sending ChangeConfig:ADD_PEER:VOTER on tablet 0d6b8fb1ba064b57b8efeaf783085c76 (attempt 1) I0916 07:55:50.777385 2233 tablet_service.cc:1066] Processing DeleteTablet for tablet 0d6b8fb1ba064b57b8efeaf783085c76 with delete_type TABLET_DATA_TOMBSTONED (TS 7c2cba5d630949b894e60e1bb10b8148 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46832 I0916 07:55:50.779500 2175 consensus_queue.cc:206] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "12336a106614452ab8811f1ca5651a7f" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 35423 } } peers { permanent_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 38109 } } peers { permanent_uuid: "7c2cba5d630949b894e60e1bb10b8148" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39907 } } I0916 07:55:50.781803 2553 ts_tablet_manager.cc:1440] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0916 07:55:50.783042 2310 raft_consensus.cc:1184] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4 [term 2 FOLLOWER]: Refusing update from remote peer 12336a106614452ab8811f1ca5651a7f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch) W0916 07:55:50.783394 2136 consensus_peers.cc:458] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f -> Peer 7c2cba5d630949b894e60e1bb10b8148 (127.0.61.2:39907): Couldn't send request to peer 7c2cba5d630949b894e60e1bb10b8148. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry. I0916 07:55:50.784229 2508 consensus_queue.cc:984] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 38109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I0916 07:55:50.788344 2515 raft_consensus.cc:2792] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 7c2cba5d630949b894e60e1bb10b8148 (127.0.61.2) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "12336a106614452ab8811f1ca5651a7f" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 35423 } } peers { permanent_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 38109 } } peers { permanent_uuid: "7c2cba5d630949b894e60e1bb10b8148" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39907 } } } I0916 07:55:50.789737 2310 raft_consensus.cc:2792] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 7c2cba5d630949b894e60e1bb10b8148 (127.0.61.2) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "12336a106614452ab8811f1ca5651a7f" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 35423 } } peers { permanent_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 38109 } } peers { permanent_uuid: "7c2cba5d630949b894e60e1bb10b8148" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39907 } } } I0916 07:55:50.791066 2553 ts_tablet_manager.cc:1453] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1 I0916 07:55:50.791368 2553 log.cc:990] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148: Deleting WAL directory at /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568620531643032-244/minicluster-data/ts-1-root/wals/0d6b8fb1ba064b57b8efeaf783085c76 I0916 07:55:50.793004 2072 catalog_manager.cc:3490] TS 7c2cba5d630949b894e60e1bb10b8148 (127.0.61.2:39907): tablet 0d6b8fb1ba064b57b8efeaf783085c76 (table test-workload [id=92a032d2b20c43bd952da7b753881252]) successfully deleted I0916 07:55:50.796012 2075 catalog_manager.cc:3654] ChangeConfig:ADD_PEER:VOTER RPC for tablet 0d6b8fb1ba064b57b8efeaf783085c76 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1) I0916 07:55:50.798111 2089 catalog_manager.cc:4120] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f reported cstate change: config changed from index 3 to 4, VOTER 7c2cba5d630949b894e60e1bb10b8148 (127.0.61.2) added. New cstate: current_term: 2 leader_uuid: "12336a106614452ab8811f1ca5651a7f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "12336a106614452ab8811f1ca5651a7f" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 35423 } } peers { permanent_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 38109 } } peers { permanent_uuid: "7c2cba5d630949b894e60e1bb10b8148" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39907 } } } I0916 07:55:51.179560 2554 ts_tablet_manager.cc:708] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148: Initiating tablet copy from peer 12336a106614452ab8811f1ca5651a7f (127.0.61.1:35423) I0916 07:55:51.180019 2554 tablet_copy_client.cc:241] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148: tablet copy: Beginning tablet copy session from remote peer at address 127.0.61.1:35423 I0916 07:55:51.181373 2185 tablet_copy_service.cc:135] P 12336a106614452ab8811f1ca5651a7f: Received BeginTabletCopySession request for tablet 0d6b8fb1ba064b57b8efeaf783085c76 from peer 7c2cba5d630949b894e60e1bb10b8148 ({username='slave'} at 127.0.0.1:52104) I0916 07:55:51.181699 2185 tablet_copy_service.cc:156] P 12336a106614452ab8811f1ca5651a7f: Beginning new tablet copy session on tablet 0d6b8fb1ba064b57b8efeaf783085c76 from peer 7c2cba5d630949b894e60e1bb10b8148 at {username='slave'} at 127.0.0.1:52104: session id = 7c2cba5d630949b894e60e1bb10b8148-0d6b8fb1ba064b57b8efeaf783085c76 I0916 07:55:51.186170 2185 tablet_copy_source_session.cc:202] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f: Tablet Copy: opened 0 blocks and 1 log segments I0916 07:55:51.188261 2554 ts_tablet_manager.cc:1440] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148: Deleting tablet data with delete state TABLET_DATA_COPYING I0916 07:55:51.196408 2554 ts_tablet_manager.cc:1453] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1 I0916 07:55:51.196909 2554 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 0d6b8fb1ba064b57b8efeaf783085c76. 1 dirs total, 0 dirs full, 0 dirs failed I0916 07:55:51.200706 2554 tablet_copy_client.cc:579] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148: tablet copy: Starting download of 0 data blocks... I0916 07:55:51.201072 2554 tablet_copy_client.cc:542] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148: tablet copy: Starting download of 1 WAL segments... I0916 07:55:51.203626 2554 tablet_copy_client.cc:418] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0916 07:55:51.208009 2554 tablet_bootstrap.cc:439] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148: Bootstrap starting. I0916 07:55:51.217855 2554 tablet_bootstrap.cc:439] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0916 07:55:51.218441 2554 tablet_bootstrap.cc:439] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148: Bootstrap complete. I0916 07:55:51.218860 2554 ts_tablet_manager.cc:1085] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148: Time spent bootstrapping tablet: real 0.011s user 0.012s sys 0.000s I0916 07:55:51.220149 2554 raft_consensus.cc:340] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "12336a106614452ab8811f1ca5651a7f" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 35423 } } peers { permanent_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 38109 } } peers { permanent_uuid: "7c2cba5d630949b894e60e1bb10b8148" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39907 } } I0916 07:55:51.220702 2554 raft_consensus.cc:705] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c2cba5d630949b894e60e1bb10b8148, State: Initialized, Role: FOLLOWER I0916 07:55:51.221175 2554 consensus_queue.cc:229] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "12336a106614452ab8811f1ca5651a7f" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 35423 } } peers { permanent_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 38109 } } peers { permanent_uuid: "7c2cba5d630949b894e60e1bb10b8148" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39907 } } I0916 07:55:51.223106 2554 ts_tablet_manager.cc:1113] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s I0916 07:55:51.224303 2185 tablet_copy_service.cc:325] P 12336a106614452ab8811f1ca5651a7f: Request end of tablet copy session 7c2cba5d630949b894e60e1bb10b8148-0d6b8fb1ba064b57b8efeaf783085c76 received from {username='slave'} at 127.0.0.1:52104 I0916 07:55:51.224661 2185 tablet_copy_service.cc:413] P 12336a106614452ab8811f1ca5651a7f: ending tablet copy session 7c2cba5d630949b894e60e1bb10b8148-0d6b8fb1ba064b57b8efeaf783085c76 on tablet 0d6b8fb1ba064b57b8efeaf783085c76 with peer 7c2cba5d630949b894e60e1bb10b8148 Master Summary UUID | Address | Status ----------------------------------+-------------------+--------- 9bd8119743184be0a7f051deead359d1 | 127.0.61.62:34307 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+------------------+---------+---------- 12336a106614452ab8811f1ca5651a7f | 127.0.61.1:35423 | HEALTHY | 7c2cba5d630949b894e60e1bb10b8148 | 127.0.61.2:39907 | HEALTHY | dec5051aad8d4ab8927d10e1a4b937b4 | 127.0.61.3:38109 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 4 server(s) checked Tablet Summary Tablet 0d6b8fb1ba064b57b8efeaf783085c76 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's 12336a106614452ab8811f1ca5651a7f (127.0.61.1:35423): RUNNING [LEADER] dec5051aad8d4ab8927d10e1a4b937b4 (127.0.61.3:38109): RUNNING 7c2cba5d630949b894e60e1bb10b8148 (127.0.61.2:39907): RUNNING All reported replicas are: A = 12336a106614452ab8811f1ca5651a7f B = dec5051aad8d4ab8927d10e1a4b937b4 C = 7c2cba5d630949b894e60e1bb10b8148 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 2 | 4 | Yes B | A* B C | 2 | 4 | Yes C | A B C | 2 | 4 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0916 07:55:51.425421 244 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0916 07:55:51.790789 2242 raft_consensus.cc:1126] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.3->[2.4-2.4] Dedup: 2.4->[] W0916 07:55:52.046422 2208 debug-util.cc:397] Leaking SignalData structure 0x7b08000266c0 after lost signal to thread 247 Master Summary UUID | Address | Status ----------------------------------+-------------------+--------- 9bd8119743184be0a7f051deead359d1 | 127.0.61.62:34307 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+------------------+---------+---------- 12336a106614452ab8811f1ca5651a7f | 127.0.61.1:35423 | HEALTHY | 7c2cba5d630949b894e60e1bb10b8148 | 127.0.61.2:39907 | HEALTHY | dec5051aad8d4ab8927d10e1a4b937b4 | 127.0.61.3:38109 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 4 server(s) checked Tablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+---------+---------------+---------+------------+------------------+------------- test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0916 07:55:52.677925 244 log_verifier.cc:121] Checking tablet 0d6b8fb1ba064b57b8efeaf783085c76 I0916 07:55:52.754932 244 log_verifier.cc:172] Verified matching terms for 4 ops in tablet 0d6b8fb1ba064b57b8efeaf783085c76 I0916 07:55:52.759182 244 tablet_server.cc:159] TabletServer@127.0.61.1:35423 shutting down... I0916 07:55:52.793138 244 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0916 07:55:52.793565 244 tablet_replica.cc:273] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f: stopping tablet replica I0916 07:55:52.794039 244 raft_consensus.cc:2147] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f [term 2 LEADER]: Raft consensus shutting down. I0916 07:55:52.795011 244 raft_consensus.cc:2174] T 0d6b8fb1ba064b57b8efeaf783085c76 P 12336a106614452ab8811f1ca5651a7f [term 2 FOLLOWER]: Raft consensus is shut down! I0916 07:55:52.818622 244 tablet_server.cc:173] TabletServer@127.0.61.1:35423 shutdown complete. I0916 07:55:52.830932 244 tablet_server.cc:159] TabletServer@127.0.61.2:39907 shutting down... I0916 07:55:52.859714 244 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0916 07:55:52.860155 244 tablet_replica.cc:273] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148: stopping tablet replica I0916 07:55:52.860642 244 raft_consensus.cc:2147] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148 [term 2 FOLLOWER]: Raft consensus shutting down. I0916 07:55:52.861024 244 raft_consensus.cc:2174] T 0d6b8fb1ba064b57b8efeaf783085c76 P 7c2cba5d630949b894e60e1bb10b8148 [term 2 FOLLOWER]: Raft consensus is shut down! I0916 07:55:53.829588 2611 raft_consensus.cc:472] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 12336a106614452ab8811f1ca5651a7f) I0916 07:55:53.830109 2611 raft_consensus.cc:494] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "12336a106614452ab8811f1ca5651a7f" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 35423 } } peers { permanent_uuid: "dec5051aad8d4ab8927d10e1a4b937b4" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 38109 } } peers { permanent_uuid: "7c2cba5d630949b894e60e1bb10b8148" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 39907 } } I0916 07:55:53.832018 2611 leader_election.cc:296] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 12336a106614452ab8811f1ca5651a7f (127.0.61.1:35423), 7c2cba5d630949b894e60e1bb10b8148 (127.0.61.2:39907) W0916 07:55:53.834908 2271 leader_election.cc:341] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7c2cba5d630949b894e60e1bb10b8148 (127.0.61.2:39907): Network error: Client connection negotiation failed: client connection to 127.0.61.2:39907: connect: Connection refused (error 111) W0916 07:55:53.835351 2274 leader_election.cc:341] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 12336a106614452ab8811f1ca5651a7f (127.0.61.1:35423): Network error: Client connection negotiation failed: client connection to 127.0.61.1:35423: connect: Connection refused (error 111) I0916 07:55:53.835608 2274 leader_election.cc:310] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dec5051aad8d4ab8927d10e1a4b937b4; no voters: 12336a106614452ab8811f1ca5651a7f, 7c2cba5d630949b894e60e1bb10b8148 I0916 07:55:53.836222 2611 raft_consensus.cc:2592] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority W0916 07:55:53.880955 244 thread.cc:499] Waited for 1000ms trying to join with diag-logger (tid 2208) I0916 07:55:54.005455 244 tablet_server.cc:173] TabletServer@127.0.61.2:39907 shutdown complete. I0916 07:55:54.014325 244 tablet_server.cc:159] TabletServer@127.0.61.3:38109 shutting down... I0916 07:55:54.033938 244 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0916 07:55:54.034251 244 tablet_replica.cc:273] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4: stopping tablet replica I0916 07:55:54.034587 244 raft_consensus.cc:2147] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4 [term 2 FOLLOWER]: Raft consensus shutting down. I0916 07:55:54.034853 244 raft_consensus.cc:2174] T 0d6b8fb1ba064b57b8efeaf783085c76 P dec5051aad8d4ab8927d10e1a4b937b4 [term 2 FOLLOWER]: Raft consensus is shut down! I0916 07:55:54.055197 244 tablet_server.cc:173] TabletServer@127.0.61.3:38109 shutdown complete. I0916 07:55:54.063201 244 master.cc:247] Master@127.0.61.62:34307 shutting down... I0916 07:55:54.079749 244 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 9bd8119743184be0a7f051deead359d1 [term 1 LEADER]: Raft consensus shutting down. I0916 07:55:54.080108 244 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 9bd8119743184be0a7f051deead359d1 [term 1 FOLLOWER]: Raft consensus is shut down! I0916 07:55:54.080335 244 tablet_replica.cc:273] T 00000000000000000000000000000000 P 9bd8119743184be0a7f051deead359d1: stopping tablet replica I0916 07:55:54.096901 244 master.cc:258] Master@127.0.61.62:34307 shutdown complete. [ OK ] FailedTabletsAreReplacedITest.OneReplica/0 (7058 ms) [ RUN ] FailedTabletsAreReplacedITest.OneReplica/1 I0916 07:55:54.117676 244 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.0.61.62:40441 I0916 07:55:54.128959 244 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 53016us I0916 07:55:54.129364 244 fs_manager.cc:263] Metadata directory not provided I0916 07:55:54.129503 244 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0916 07:55:54.129725 244 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0916 07:55:54.129832 244 server_base.cc:453] Attempting to create new FS layout instead I0916 07:55:54.130861 244 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568620531643032-244/minicluster-data/master-0-root/instance: uuid: "490537774c7b411291a492c7d78633ca" format_stamp: "Formatted at 2019-09-16 07:55:54 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:54.138525 244 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s I0916 07:55:54.139922 244 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0916 07:55:54.146097 244 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0916 07:55:54.146258 244 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568620531643032-244/minicluster-data/master-0-root uuid: "490537774c7b411291a492c7d78633ca" format_stamp: "Formatted at 2019-09-16 07:55:54 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:54.146495 244 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568620531643032-244/minicluster-data/master-0-root metadata directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568620531643032-244/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568620531643032-244/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0916 07:55:54.233434 244 rpc_server.cc:205] RPC server started. Bound to: 127.0.61.62:40441 I0916 07:55:54.235199 244 webserver.cc:374] Webserver started at http://127.0.61.62:44961/ using document root and password file I0916 07:55:54.236918 2672 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0916 07:55:54.245079 2672 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 490537774c7b411291a492c7d78633ca: Bootstrap starting. I0916 07:55:54.247738 2672 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 490537774c7b411291a492c7d78633ca: No blocks or log segments found. Creating new log. I0916 07:55:54.250892 2672 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 490537774c7b411291a492c7d78633ca: No bootstrap required, opened a new log I0916 07:55:54.252326 2672 raft_consensus.cc:340] T 00000000000000000000000000000000 P 490537774c7b411291a492c7d78633ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "490537774c7b411291a492c7d78633ca" member_type: VOTER } I0916 07:55:54.252730 2672 raft_consensus.cc:366] T 00000000000000000000000000000000 P 490537774c7b411291a492c7d78633ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0916 07:55:54.252882 2672 raft_consensus.cc:705] T 00000000000000000000000000000000 P 490537774c7b411291a492c7d78633ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 490537774c7b411291a492c7d78633ca, State: Initialized, Role: FOLLOWER I0916 07:55:54.253289 2672 consensus_queue.cc:229] T 00000000000000000000000000000000 P 490537774c7b411291a492c7d78633ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "490537774c7b411291a492c7d78633ca" member_type: VOTER } I0916 07:55:54.253660 2672 raft_consensus.cc:380] T 00000000000000000000000000000000 P 490537774c7b411291a492c7d78633ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0916 07:55:54.253808 2672 raft_consensus.cc:472] T 00000000000000000000000000000000 P 490537774c7b411291a492c7d78633ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0916 07:55:54.253979 2672 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 490537774c7b411291a492c7d78633ca [term 0 FOLLOWER]: Advancing to term 1 I0916 07:55:54.257577 2672 raft_consensus.cc:494] T 00000000000000000000000000000000 P 490537774c7b411291a492c7d78633ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "490537774c7b411291a492c7d78633ca" member_type: VOTER } I0916 07:55:54.257972 2672 leader_election.cc:310] T 00000000000000000000000000000000 P 490537774c7b411291a492c7d78633ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 490537774c7b411291a492c7d78633ca; no voters: I0916 07:55:54.259297 2672 leader_election.cc:296] T 00000000000000000000000000000000 P 490537774c7b411291a492c7d78633ca [CANDIDATE]: Term 1 election: Requested vote from peers I0916 07:55:54.259598 2674 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 490537774c7b411291a492c7d78633ca [term 1 FOLLOWER]: Leader election won for term 1 I0916 07:55:54.261178 2674 raft_consensus.cc:667] T 00000000000000000000000000000000 P 490537774c7b411291a492c7d78633ca [term 1 LEADER]: Becoming Leader. State: Replica: 490537774c7b411291a492c7d78633ca, State: Running, Role: LEADER I0916 07:55:54.261723 2674 consensus_queue.cc:206] T 00000000000000000000000000000000 P 490537774c7b411291a492c7d78633ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "490537774c7b411291a492c7d78633ca" member_type: VOTER } I0916 07:55:54.264586 2676 sys_catalog.cc:340] T 00000000000000000000000000000000 P 490537774c7b411291a492c7d78633ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 490537774c7b411291a492c7d78633ca. Latest consensus state: current_term: 1 leader_uuid: "490537774c7b411291a492c7d78633ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "490537774c7b411291a492c7d78633ca" member_type: VOTER } } I0916 07:55:54.265075 2676 sys_catalog.cc:343] T 00000000000000000000000000000000 P 490537774c7b411291a492c7d78633ca [sys.catalog]: This master's current role is: LEADER I0916 07:55:54.264770 2672 sys_catalog.cc:340] T 00000000000000000000000000000000 P 490537774c7b411291a492c7d78633ca [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "490537774c7b411291a492c7d78633ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "490537774c7b411291a492c7d78633ca" member_type: VOTER } } I0916 07:55:54.265481 2672 sys_catalog.cc:343] T 00000000000000000000000000000000 P 490537774c7b411291a492c7d78633ca [sys.catalog]: This master's current role is: LEADER I0916 07:55:54.265483 2675 sys_catalog.cc:340] T 00000000000000000000000000000000 P 490537774c7b411291a492c7d78633ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "490537774c7b411291a492c7d78633ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "490537774c7b411291a492c7d78633ca" member_type: VOTER } } I0916 07:55:54.266703 2675 sys_catalog.cc:343] T 00000000000000000000000000000000 P 490537774c7b411291a492c7d78633ca [sys.catalog]: This master's current role is: LEADER I0916 07:55:54.267530 2672 sys_catalog.cc:428] T 00000000000000000000000000000000 P 490537774c7b411291a492c7d78633ca [sys.catalog]: configured and running, proceeding with master startup. I0916 07:55:54.267829 2679 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0916 07:55:54.268808 244 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0916 07:55:54.270464 2679 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0916 07:55:54.299007 2679 catalog_manager.cc:981] Generated new certificate authority record I0916 07:55:54.300981 2679 catalog_manager.cc:1106] Loading token signing keys... I0916 07:55:54.310140 2679 catalog_manager.cc:4435] T 00000000000000000000000000000000 P 490537774c7b411291a492c7d78633ca: Generated new TSK 0 I0916 07:55:54.345355 244 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 53016us I0916 07:55:54.345695 244 fs_manager.cc:263] Metadata directory not provided I0916 07:55:54.345815 244 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0916 07:55:54.345973 244 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0916 07:55:54.346068 244 server_base.cc:453] Attempting to create new FS layout instead I0916 07:55:54.346930 244 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568620531643032-244/minicluster-data/ts-0-root/instance: uuid: "0938c74cab11424fa5bc15e4cba13157" format_stamp: "Formatted at 2019-09-16 07:55:54 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:54.354771 244 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.005s sys 0.005s I0916 07:55:54.356408 244 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0916 07:55:54.362663 244 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0916 07:55:54.362834 244 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568620531643032-244/minicluster-data/ts-0-root uuid: "0938c74cab11424fa5bc15e4cba13157" format_stamp: "Formatted at 2019-09-16 07:55:54 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:54.363054 244 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568620531643032-244/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568620531643032-244/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568620531643032-244/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0916 07:55:54.428589 244 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0916 07:55:54.428741 244 ts_tablet_manager.cc:403] Registered 0 tablets I0916 07:55:54.467787 244 rpc_server.cc:205] RPC server started. Bound to: 127.0.61.1:42243 I0916 07:55:54.469557 244 webserver.cc:374] Webserver started at http://127.0.61.1:46477/ using document root and password file I0916 07:55:54.486675 244 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 53016us I0916 07:55:54.487115 244 fs_manager.cc:263] Metadata directory not provided I0916 07:55:54.487277 244 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0916 07:55:54.487498 244 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0916 07:55:54.487649 244 server_base.cc:453] Attempting to create new FS layout instead I0916 07:55:54.488826 244 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568620531643032-244/minicluster-data/ts-1-root/instance: uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" format_stamp: "Formatted at 2019-09-16 07:55:54 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:54.489754 2746 heartbeater.cc:322] Connected to a master server at 127.0.61.62:40441 I0916 07:55:54.490062 2746 heartbeater.cc:406] Registering TS with master... I0916 07:55:54.490484 2746 heartbeater.cc:455] Master 127.0.61.62:40441 requested a full tablet report, sending... I0916 07:55:54.492156 2639 ts_manager.cc:161] Registered new tserver with Master: 0938c74cab11424fa5bc15e4cba13157 (127.0.61.1:42243) I0916 07:55:54.494679 2639 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46930 I0916 07:55:54.499094 244 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.007s sys 0.003s I0916 07:55:54.500613 244 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0916 07:55:54.507166 244 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0916 07:55:54.507341 244 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568620531643032-244/minicluster-data/ts-1-root uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" format_stamp: "Formatted at 2019-09-16 07:55:54 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:54.507582 244 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568620531643032-244/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568620531643032-244/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568620531643032-244/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0916 07:55:54.542719 244 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0916 07:55:54.542891 244 ts_tablet_manager.cc:403] Registered 0 tablets I0916 07:55:54.579285 244 rpc_server.cc:205] RPC server started. Bound to: 127.0.61.2:33603 I0916 07:55:54.580976 244 webserver.cc:374] Webserver started at http://127.0.61.2:44759/ using document root and password file I0916 07:55:54.595118 2814 heartbeater.cc:322] Connected to a master server at 127.0.61.62:40441 I0916 07:55:54.595409 2814 heartbeater.cc:406] Registering TS with master... I0916 07:55:54.595831 2814 heartbeater.cc:455] Master 127.0.61.62:40441 requested a full tablet report, sending... I0916 07:55:54.597306 2639 ts_manager.cc:161] Registered new tserver with Master: f2a7d0ddeb8343839dd8dc3f3413a783 (127.0.61.2:33603) I0916 07:55:54.598579 2639 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46932 I0916 07:55:54.600910 244 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 53016us I0916 07:55:54.601260 244 fs_manager.cc:263] Metadata directory not provided I0916 07:55:54.601402 244 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0916 07:55:54.601588 244 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0916 07:55:54.601693 244 server_base.cc:453] Attempting to create new FS layout instead I0916 07:55:54.602578 244 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568620531643032-244/minicluster-data/ts-2-root/instance: uuid: "32da123829d846f2b8e23df043e04a15" format_stamp: "Formatted at 2019-09-16 07:55:54 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:54.611220 244 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.005s sys 0.005s I0916 07:55:54.612641 244 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0916 07:55:54.619874 244 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0916 07:55:54.620064 244 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568620531643032-244/minicluster-data/ts-2-root uuid: "32da123829d846f2b8e23df043e04a15" format_stamp: "Formatted at 2019-09-16 07:55:54 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:54.620307 244 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568620531643032-244/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568620531643032-244/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568620531643032-244/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0916 07:55:54.666321 244 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0916 07:55:54.666520 244 ts_tablet_manager.cc:403] Registered 0 tablets I0916 07:55:54.703336 244 rpc_server.cc:205] RPC server started. Bound to: 127.0.61.3:44291 I0916 07:55:54.705108 244 webserver.cc:374] Webserver started at http://127.0.61.3:46083/ using document root and password file I0916 07:55:54.717914 2881 heartbeater.cc:322] Connected to a master server at 127.0.61.62:40441 I0916 07:55:54.718209 2881 heartbeater.cc:406] Registering TS with master... I0916 07:55:54.718631 2881 heartbeater.cc:455] Master 127.0.61.62:40441 requested a full tablet report, sending... I0916 07:55:54.720145 2639 ts_manager.cc:161] Registered new tserver with Master: 32da123829d846f2b8e23df043e04a15 (127.0.61.3:44291) I0916 07:55:54.721534 2639 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46934 I0916 07:55:54.725010 244 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 53016us I0916 07:55:54.725368 244 fs_manager.cc:263] Metadata directory not provided I0916 07:55:54.725522 244 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0916 07:55:54.725698 244 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0916 07:55:54.725802 244 server_base.cc:453] Attempting to create new FS layout instead I0916 07:55:54.726776 244 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568620531643032-244/minicluster-data/ts-3-root/instance: uuid: "ca39e2b29dd34cc2a249e27cd9590c4e" format_stamp: "Formatted at 2019-09-16 07:55:54 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:54.736069 244 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.010s sys 0.001s I0916 07:55:54.737586 244 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0916 07:55:54.745651 244 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s I0916 07:55:54.745856 244 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568620531643032-244/minicluster-data/ts-3-root uuid: "ca39e2b29dd34cc2a249e27cd9590c4e" format_stamp: "Formatted at 2019-09-16 07:55:54 on dist-test-slave-dist-test-slave-p3b0" I0916 07:55:54.746083 244 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568620531643032-244/minicluster-data/ts-3-root metadata directory: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568620531643032-244/minicluster-data/ts-3-root 1 data directories: /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568620531643032-244/minicluster-data/ts-3-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0916 07:55:54.773377 244 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0916 07:55:54.773581 244 ts_tablet_manager.cc:403] Registered 0 tablets I0916 07:55:54.811287 244 rpc_server.cc:205] RPC server started. Bound to: 127.0.61.4:39741 I0916 07:55:54.813185 244 webserver.cc:374] Webserver started at http://127.0.61.4:38077/ using document root and password file I0916 07:55:54.825356 2948 heartbeater.cc:322] Connected to a master server at 127.0.61.62:40441 I0916 07:55:54.825690 2948 heartbeater.cc:406] Registering TS with master... I0916 07:55:54.826205 2948 heartbeater.cc:455] Master 127.0.61.62:40441 requested a full tablet report, sending... I0916 07:55:54.827685 2639 ts_manager.cc:161] Registered new tserver with Master: ca39e2b29dd34cc2a249e27cd9590c4e (127.0.61.4:39741) I0916 07:55:54.828682 244 internal_mini_cluster.cc:328] 4 TS(s) registered with all masters after 0.013606798s I0916 07:55:54.829305 2639 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46936 I0916 07:55:54.848835 244 test_util.cc:212] Using random seed: -285898199 I0916 07:55:54.869300 2639 catalog_manager.cc:1440] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46940: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0916 07:55:54.911267 2713 tablet_service.cc:1022] Processing CreateTablet for tablet 3485e0e47b994bf892b67638ce21a806 (table=test-workload [id=81c4fe0d20654d429dedb2035fe0c0a4]), partition=RANGE (key) PARTITION UNBOUNDED I0916 07:55:54.912243 2713 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 3485e0e47b994bf892b67638ce21a806. 1 dirs total, 0 dirs full, 0 dirs failed I0916 07:55:54.914419 2848 tablet_service.cc:1022] Processing CreateTablet for tablet 3485e0e47b994bf892b67638ce21a806 (table=test-workload [id=81c4fe0d20654d429dedb2035fe0c0a4]), partition=RANGE (key) PARTITION UNBOUNDED I0916 07:55:54.915174 2848 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 3485e0e47b994bf892b67638ce21a806. 1 dirs total, 0 dirs full, 0 dirs failed I0916 07:55:54.918112 2781 tablet_service.cc:1022] Processing CreateTablet for tablet 3485e0e47b994bf892b67638ce21a806 (table=test-workload [id=81c4fe0d20654d429dedb2035fe0c0a4]), partition=RANGE (key) PARTITION UNBOUNDED I0916 07:55:54.919029 2781 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 3485e0e47b994bf892b67638ce21a806. 1 dirs total, 0 dirs full, 0 dirs failed I0916 07:55:54.925779 2971 tablet_bootstrap.cc:439] T 3485e0e47b994bf892b67638ce21a806 P 0938c74cab11424fa5bc15e4cba13157: Bootstrap starting. I0916 07:55:54.928866 2971 tablet_bootstrap.cc:590] T 3485e0e47b994bf892b67638ce21a806 P 0938c74cab11424fa5bc15e4cba13157: No blocks or log segments found. Creating new log. I0916 07:55:54.933629 2973 tablet_bootstrap.cc:439] T 3485e0e47b994bf892b67638ce21a806 P 32da123829d846f2b8e23df043e04a15: Bootstrap starting. I0916 07:55:54.933821 2972 tablet_bootstrap.cc:439] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783: Bootstrap starting. I0916 07:55:54.936327 2973 tablet_bootstrap.cc:590] T 3485e0e47b994bf892b67638ce21a806 P 32da123829d846f2b8e23df043e04a15: No blocks or log segments found. Creating new log. I0916 07:55:54.938097 2972 tablet_bootstrap.cc:590] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783: No blocks or log segments found. Creating new log. I0916 07:55:54.940717 2973 tablet_bootstrap.cc:439] T 3485e0e47b994bf892b67638ce21a806 P 32da123829d846f2b8e23df043e04a15: No bootstrap required, opened a new log I0916 07:55:54.940871 2971 tablet_bootstrap.cc:439] T 3485e0e47b994bf892b67638ce21a806 P 0938c74cab11424fa5bc15e4cba13157: No bootstrap required, opened a new log I0916 07:55:54.941200 2973 ts_tablet_manager.cc:1085] T 3485e0e47b994bf892b67638ce21a806 P 32da123829d846f2b8e23df043e04a15: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s I0916 07:55:54.941431 2971 ts_tablet_manager.cc:1085] T 3485e0e47b994bf892b67638ce21a806 P 0938c74cab11424fa5bc15e4cba13157: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.009s I0916 07:55:54.942631 2973 raft_consensus.cc:340] T 3485e0e47b994bf892b67638ce21a806 P 32da123829d846f2b8e23df043e04a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32da123829d846f2b8e23df043e04a15" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 44291 } } peers { permanent_uuid: "0938c74cab11424fa5bc15e4cba13157" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 42243 } } peers { permanent_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 33603 } } I0916 07:55:54.943152 2973 raft_consensus.cc:366] T 3485e0e47b994bf892b67638ce21a806 P 32da123829d846f2b8e23df043e04a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0916 07:55:54.943322 2973 raft_consensus.cc:705] T 3485e0e47b994bf892b67638ce21a806 P 32da123829d846f2b8e23df043e04a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32da123829d846f2b8e23df043e04a15, State: Initialized, Role: FOLLOWER I0916 07:55:54.943763 2973 consensus_queue.cc:229] T 3485e0e47b994bf892b67638ce21a806 P 32da123829d846f2b8e23df043e04a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32da123829d846f2b8e23df043e04a15" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 44291 } } peers { permanent_uuid: "0938c74cab11424fa5bc15e4cba13157" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 42243 } } peers { permanent_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 33603 } } I0916 07:55:54.943600 2971 raft_consensus.cc:340] T 3485e0e47b994bf892b67638ce21a806 P 0938c74cab11424fa5bc15e4cba13157 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32da123829d846f2b8e23df043e04a15" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 44291 } } peers { permanent_uuid: "0938c74cab11424fa5bc15e4cba13157" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 42243 } } peers { permanent_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 33603 } } I0916 07:55:54.944810 2971 raft_consensus.cc:366] T 3485e0e47b994bf892b67638ce21a806 P 0938c74cab11424fa5bc15e4cba13157 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0916 07:55:54.945201 2971 raft_consensus.cc:705] T 3485e0e47b994bf892b67638ce21a806 P 0938c74cab11424fa5bc15e4cba13157 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0938c74cab11424fa5bc15e4cba13157, State: Initialized, Role: FOLLOWER I0916 07:55:54.946066 2881 heartbeater.cc:447] Master 127.0.61.62:40441 was elected leader, sending a full tablet report... I0916 07:55:54.946096 2971 consensus_queue.cc:229] T 3485e0e47b994bf892b67638ce21a806 P 0938c74cab11424fa5bc15e4cba13157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32da123829d846f2b8e23df043e04a15" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 44291 } } peers { permanent_uuid: "0938c74cab11424fa5bc15e4cba13157" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 42243 } } peers { permanent_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 33603 } } I0916 07:55:54.947000 2972 tablet_bootstrap.cc:439] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783: No bootstrap required, opened a new log I0916 07:55:54.947526 2972 ts_tablet_manager.cc:1085] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s I0916 07:55:54.949934 2746 heartbeater.cc:447] Master 127.0.61.62:40441 was elected leader, sending a full tablet report... I0916 07:55:54.949807 2972 raft_consensus.cc:340] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32da123829d846f2b8e23df043e04a15" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 44291 } } peers { permanent_uuid: "0938c74cab11424fa5bc15e4cba13157" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 42243 } } peers { permanent_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 33603 } } I0916 07:55:54.950539 2972 raft_consensus.cc:366] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0916 07:55:54.950793 2972 raft_consensus.cc:705] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2a7d0ddeb8343839dd8dc3f3413a783, State: Initialized, Role: FOLLOWER I0916 07:55:54.951210 2971 ts_tablet_manager.cc:1113] T 3485e0e47b994bf892b67638ce21a806 P 0938c74cab11424fa5bc15e4cba13157: Time spent starting tablet: real 0.009s user 0.002s sys 0.006s I0916 07:55:54.951468 2972 consensus_queue.cc:229] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32da123829d846f2b8e23df043e04a15" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 44291 } } peers { permanent_uuid: "0938c74cab11424fa5bc15e4cba13157" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 42243 } } peers { permanent_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 33603 } } I0916 07:55:54.952399 2973 ts_tablet_manager.cc:1113] T 3485e0e47b994bf892b67638ce21a806 P 32da123829d846f2b8e23df043e04a15: Time spent starting tablet: real 0.011s user 0.000s sys 0.005s I0916 07:55:54.954560 2814 heartbeater.cc:447] Master 127.0.61.62:40441 was elected leader, sending a full tablet report... I0916 07:55:54.955716 2972 ts_tablet_manager.cc:1113] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s I0916 07:55:55.078742 2979 raft_consensus.cc:472] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0916 07:55:55.079181 2979 raft_consensus.cc:494] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32da123829d846f2b8e23df043e04a15" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 44291 } } peers { permanent_uuid: "0938c74cab11424fa5bc15e4cba13157" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 42243 } } peers { permanent_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 33603 } } I0916 07:55:55.080546 2979 leader_election.cc:296] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32da123829d846f2b8e23df043e04a15 (127.0.61.3:44291), 0938c74cab11424fa5bc15e4cba13157 (127.0.61.1:42243) I0916 07:55:55.093389 2723 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "3485e0e47b994bf892b67638ce21a806" candidate_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0938c74cab11424fa5bc15e4cba13157" is_pre_election: true I0916 07:55:55.094190 2723 raft_consensus.cc:2350] T 3485e0e47b994bf892b67638ce21a806 P 0938c74cab11424fa5bc15e4cba13157 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2a7d0ddeb8343839dd8dc3f3413a783 in term 0. I0916 07:55:55.094244 2858 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "3485e0e47b994bf892b67638ce21a806" candidate_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32da123829d846f2b8e23df043e04a15" is_pre_election: true I0916 07:55:55.094844 2858 raft_consensus.cc:2350] T 3485e0e47b994bf892b67638ce21a806 P 32da123829d846f2b8e23df043e04a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2a7d0ddeb8343839dd8dc3f3413a783 in term 0. I0916 07:55:55.095268 2754 leader_election.cc:310] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0938c74cab11424fa5bc15e4cba13157, f2a7d0ddeb8343839dd8dc3f3413a783; no voters: I0916 07:55:55.096091 2979 raft_consensus.cc:2644] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0916 07:55:55.096323 2979 raft_consensus.cc:472] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0916 07:55:55.096551 2979 raft_consensus.cc:2890] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 [term 0 FOLLOWER]: Advancing to term 1 I0916 07:55:55.115792 2979 raft_consensus.cc:494] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32da123829d846f2b8e23df043e04a15" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 44291 } } peers { permanent_uuid: "0938c74cab11424fa5bc15e4cba13157" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 42243 } } peers { permanent_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 33603 } } I0916 07:55:55.117175 2979 leader_election.cc:296] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 [CANDIDATE]: Term 1 election: Requested vote from peers 32da123829d846f2b8e23df043e04a15 (127.0.61.3:44291), 0938c74cab11424fa5bc15e4cba13157 (127.0.61.1:42243) I0916 07:55:55.118026 2858 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "3485e0e47b994bf892b67638ce21a806" candidate_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32da123829d846f2b8e23df043e04a15" I0916 07:55:55.118063 2723 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "3485e0e47b994bf892b67638ce21a806" candidate_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0938c74cab11424fa5bc15e4cba13157" I0916 07:55:55.118623 2858 raft_consensus.cc:2890] T 3485e0e47b994bf892b67638ce21a806 P 32da123829d846f2b8e23df043e04a15 [term 0 FOLLOWER]: Advancing to term 1 I0916 07:55:55.118723 2723 raft_consensus.cc:2890] T 3485e0e47b994bf892b67638ce21a806 P 0938c74cab11424fa5bc15e4cba13157 [term 0 FOLLOWER]: Advancing to term 1 I0916 07:55:55.130677 2723 raft_consensus.cc:2350] T 3485e0e47b994bf892b67638ce21a806 P 0938c74cab11424fa5bc15e4cba13157 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2a7d0ddeb8343839dd8dc3f3413a783 in term 1. I0916 07:55:55.130683 2858 raft_consensus.cc:2350] T 3485e0e47b994bf892b67638ce21a806 P 32da123829d846f2b8e23df043e04a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2a7d0ddeb8343839dd8dc3f3413a783 in term 1. I0916 07:55:55.131960 2753 leader_election.cc:310] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32da123829d846f2b8e23df043e04a15, f2a7d0ddeb8343839dd8dc3f3413a783; no voters: I0916 07:55:55.132725 2979 raft_consensus.cc:2644] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 [term 1 FOLLOWER]: Leader election won for term 1 I0916 07:55:55.133891 2979 raft_consensus.cc:667] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 [term 1 LEADER]: Becoming Leader. State: Replica: f2a7d0ddeb8343839dd8dc3f3413a783, State: Running, Role: LEADER I0916 07:55:55.134519 2979 consensus_queue.cc:206] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32da123829d846f2b8e23df043e04a15" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 44291 } } peers { permanent_uuid: "0938c74cab11424fa5bc15e4cba13157" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 42243 } } peers { permanent_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 33603 } } I0916 07:55:55.139703 2638 catalog_manager.cc:4120] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 reported cstate change: term changed from 0 to 1, leader changed from to f2a7d0ddeb8343839dd8dc3f3413a783 (127.0.61.2). New cstate: current_term: 1 leader_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32da123829d846f2b8e23df043e04a15" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 44291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0938c74cab11424fa5bc15e4cba13157" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 42243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 33603 } health_report { overall_health: HEALTHY } } } Master Summary UUID | Address | Status ----------------------------------+-------------------+--------- 490537774c7b411291a492c7d78633ca | 127.0.61.62:40441 | HEALTHY Flag | Value | Tags | Master ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+------------------+---------+---------- 0938c74cab11424fa5bc15e4cba13157 | 127.0.61.1:42243 | HEALTHY | 32da123829d846f2b8e23df043e04a15 | 127.0.61.3:44291 | HEALTHY | ca39e2b29dd34cc2a249e27cd9590c4e | 127.0.61.4:39741 | HEALTHY | f2a7d0ddeb8343839dd8dc3f3413a783 | 127.0.61.2:33603 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet 3485e0e47b994bf892b67638ce21a806 of table 'test-workload' is conflicted: 5 replicas' active configs disagree with the leader master's 32da123829d846f2b8e23df043e04a15 (127.0.61.3:44291): RUNNING 0938c74cab11424fa5bc15e4cba13157 (127.0.61.1:42243): RUNNING f2a7d0ddeb8343839dd8dc3f3413a783 (127.0.61.2:33603): RUNNING [LEADER] All reported replicas are: A = 32da123829d846f2b8e23df043e04a15 B = 0938c74cab11424fa5bc15e4cba13157 C = f2a7d0ddeb8343839dd8dc3f3413a783 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C | 1 | -1 | Yes B | A B C | 1 | -1 | Yes C | A B C* | 1 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 0 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+------------------+--------------- Small | ca39e2b29dd34cc2a249e27cd9590c4e | 127.0.61.4:39741 | 0 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0916 07:55:55.353369 244 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0916 07:55:55.572670 2979 consensus_queue.cc:984] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0938c74cab11424fa5bc15e4cba13157" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0916 07:55:55.594838 2981 consensus_queue.cc:984] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32da123829d846f2b8e23df043e04a15" member_type: VOTER last_known_addr { host: "127.0.61.3" port: 44291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s Master Summary UUID | Address | Status ----------------------------------+-------------------+--------- 490537774c7b411291a492c7d78633ca | 127.0.61.62:40441 | HEALTHY Flag | Value | Tags | Master ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+------------------+---------+---------- 0938c74cab11424fa5bc15e4cba13157 | 127.0.61.1:42243 | HEALTHY | 32da123829d846f2b8e23df043e04a15 | 127.0.61.3:44291 | HEALTHY | ca39e2b29dd34cc2a249e27cd9590c4e | 127.0.61.4:39741 | HEALTHY | f2a7d0ddeb8343839dd8dc3f3413a783 | 127.0.61.2:33603 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 5 server(s) checked Tablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+---------+---------------+---------+------------+------------------+------------- test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 0 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+------------------+--------------- Small | ca39e2b29dd34cc2a249e27cd9590c4e | 127.0.61.4:39741 | 0 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0916 07:55:55.708959 244 log_verifier.cc:121] Checking tablet 3485e0e47b994bf892b67638ce21a806 I0916 07:55:55.777169 244 log_verifier.cc:172] Verified matching terms for 1 ops in tablet 3485e0e47b994bf892b67638ce21a806 I0916 07:55:55.777575 244 tablet_replica.cc:273] T 3485e0e47b994bf892b67638ce21a806 P 32da123829d846f2b8e23df043e04a15: stopping tablet replica I0916 07:55:55.777935 244 raft_consensus.cc:2147] T 3485e0e47b994bf892b67638ce21a806 P 32da123829d846f2b8e23df043e04a15 [term 1 FOLLOWER]: Raft consensus shutting down. I0916 07:55:55.778215 244 raft_consensus.cc:2174] T 3485e0e47b994bf892b67638ce21a806 P 32da123829d846f2b8e23df043e04a15 [term 1 FOLLOWER]: Raft consensus is shut down! I0916 07:55:55.832147 2948 heartbeater.cc:447] Master 127.0.61.62:40441 was elected leader, sending a full tablet report... Master Summary UUID | Address | Status ----------------------------------+-------------------+--------- 490537774c7b411291a492c7d78633ca | 127.0.61.62:40441 | HEALTHY Flag | Value | Tags | Master ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+------------------+---------+---------- 0938c74cab11424fa5bc15e4cba13157 | 127.0.61.1:42243 | HEALTHY | 32da123829d846f2b8e23df043e04a15 | 127.0.61.3:44291 | HEALTHY | ca39e2b29dd34cc2a249e27cd9590c4e | 127.0.61.4:39741 | HEALTHY | f2a7d0ddeb8343839dd8dc3f3413a783 | 127.0.61.2:33603 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet 3485e0e47b994bf892b67638ce21a806 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 32da123829d846f2b8e23df043e04a15 (127.0.61.3:44291): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: INJECTED ERROR: tablet failed 0938c74cab11424fa5bc15e4cba13157 (127.0.61.1:42243): RUNNING f2a7d0ddeb8343839dd8dc3f3413a783 (127.0.61.2:33603): RUNNING [LEADER] All reported replicas are: A = 32da123829d846f2b8e23df043e04a15 B = 0938c74cab11424fa5bc15e4cba13157 C = f2a7d0ddeb8343839dd8dc3f3413a783 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | [config not available] | | | B | A B C* | 1 | -1 | Yes C | A B C* | 1 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 0 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+------------------+--------------- Small | ca39e2b29dd34cc2a249e27cd9590c4e | 127.0.61.4:39741 | 0 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0916 07:55:55.953938 244 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0916 07:55:56.091964 2753 consensus_queue.cc:544] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 [LEADER]: The tablet replica hosted on peer 32da123829d846f2b8e23df043e04a15 has failed W0916 07:55:56.092566 2753 consensus_peers.cc:458] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 -> Peer 32da123829d846f2b8e23df043e04a15 (127.0.61.3:44291): Couldn't send request to peer 32da123829d846f2b8e23df043e04a15. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry. I0916 07:55:56.096771 2639 catalog_manager.cc:3871] Sending ChangeConfig:REMOVE_PEER on tablet 3485e0e47b994bf892b67638ce21a806 (attempt 1) I0916 07:55:56.099017 2791 consensus_queue.cc:206] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0938c74cab11424fa5bc15e4cba13157" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 42243 } } peers { permanent_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 33603 } } I0916 07:55:56.101853 2722 raft_consensus.cc:1184] T 3485e0e47b994bf892b67638ce21a806 P 0938c74cab11424fa5bc15e4cba13157 [term 1 FOLLOWER]: Refusing update from remote peer f2a7d0ddeb8343839dd8dc3f3413a783: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0916 07:55:56.103242 2981 consensus_queue.cc:984] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0938c74cab11424fa5bc15e4cba13157" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0916 07:55:56.107538 3032 raft_consensus.cc:2792] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 32da123829d846f2b8e23df043e04a15 (127.0.61.3) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0938c74cab11424fa5bc15e4cba13157" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 42243 } } peers { permanent_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 33603 } } } I0916 07:55:56.109077 2722 raft_consensus.cc:2792] T 3485e0e47b994bf892b67638ce21a806 P 0938c74cab11424fa5bc15e4cba13157 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 32da123829d846f2b8e23df043e04a15 (127.0.61.3) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0938c74cab11424fa5bc15e4cba13157" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 42243 } } peers { permanent_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 33603 } } } W0916 07:55:57.063683 3032 consensus_meta.cc:220] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783: Time spent flushing consensus metadata: real 0.955s user 0.005s sys 0.000s W0916 07:55:57.063683 2722 consensus_meta.cc:220] T 3485e0e47b994bf892b67638ce21a806 P 0938c74cab11424fa5bc15e4cba13157: Time spent flushing consensus metadata: real 0.953s user 0.006s sys 0.000s I0916 07:55:57.070683 2622 catalog_manager.cc:3654] ChangeConfig:REMOVE_PEER RPC for tablet 3485e0e47b994bf892b67638ce21a806 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0916 07:55:57.070683 2639 catalog_manager.cc:4120] T 3485e0e47b994bf892b67638ce21a806 P 0938c74cab11424fa5bc15e4cba13157 reported cstate change: config changed from index -1 to 2, VOTER 32da123829d846f2b8e23df043e04a15 (127.0.61.3) evicted. New cstate: current_term: 1 leader_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0938c74cab11424fa5bc15e4cba13157" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 42243 } } peers { permanent_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 33603 } } } I0916 07:55:57.085479 2639 catalog_manager.cc:3512] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 3485e0e47b994bf892b67638ce21a806 on 32da123829d846f2b8e23df043e04a15 (127.0.61.3:44291) (TS 32da123829d846f2b8e23df043e04a15 not found in new config with opid_index 2) I0916 07:55:57.086165 2638 catalog_manager.cc:3739] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 3485e0e47b994bf892b67638ce21a806 (attempt 1) I0916 07:55:57.087993 2848 tablet_service.cc:1066] Processing DeleteTablet for tablet 3485e0e47b994bf892b67638ce21a806 with delete_type TABLET_DATA_TOMBSTONED (TS 32da123829d846f2b8e23df043e04a15 not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:58446 I0916 07:55:57.090015 2791 consensus_queue.cc:206] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0938c74cab11424fa5bc15e4cba13157" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 42243 } } peers { permanent_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 33603 } } peers { permanent_uuid: "ca39e2b29dd34cc2a249e27cd9590c4e" member_type: NON_VOTER last_known_addr { host: "127.0.61.4" port: 39741 } attrs { promote: true } } I0916 07:55:57.097108 3098 ts_tablet_manager.cc:1440] T 3485e0e47b994bf892b67638ce21a806 P 32da123829d846f2b8e23df043e04a15: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0916 07:55:57.097301 2638 catalog_manager.cc:3739] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 3485e0e47b994bf892b67638ce21a806 (attempt 1) I0916 07:55:57.110136 2622 catalog_manager.cc:3671] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3485e0e47b994bf892b67638ce21a806 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER failed with leader f2a7d0ddeb8343839dd8dc3f3413a783 (127.0.61.2:33603) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time. Committed config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0938c74cab11424fa5bc15e4cba13157" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 42243 } } peers { permanent_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 33603 } }. Pending config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0938c74cab11424fa5bc15e4cba13157" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 42243 } } peers { permanent_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 33603 } } peers { permanent_uuid: "ca39e2b29dd34cc2a249e27cd9590c4e" member_type: NON_VOTER last_known_addr { host: "127.0.61.4" port: 39741 } attrs { promote: true } } I0916 07:55:57.110409 2723 raft_consensus.cc:1184] T 3485e0e47b994bf892b67638ce21a806 P 0938c74cab11424fa5bc15e4cba13157 [term 1 FOLLOWER]: Refusing update from remote peer f2a7d0ddeb8343839dd8dc3f3413a783: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I0916 07:55:57.110707 2622 catalog_manager.cc:3287] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3485e0e47b994bf892b67638ce21a806 with cas_config_opid_index 2 with a delay of 57 ms (attempt = 1) I0916 07:55:57.112162 3032 consensus_queue.cc:984] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0938c74cab11424fa5bc15e4cba13157" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0916 07:55:57.118235 3091 raft_consensus.cc:2792] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER ca39e2b29dd34cc2a249e27cd9590c4e (127.0.61.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0938c74cab11424fa5bc15e4cba13157" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 42243 } } peers { permanent_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 33603 } } peers { permanent_uuid: "ca39e2b29dd34cc2a249e27cd9590c4e" member_type: NON_VOTER last_known_addr { host: "127.0.61.4" port: 39741 } attrs { promote: true } } } I0916 07:55:57.120575 2723 raft_consensus.cc:2792] T 3485e0e47b994bf892b67638ce21a806 P 0938c74cab11424fa5bc15e4cba13157 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER ca39e2b29dd34cc2a249e27cd9590c4e (127.0.61.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0938c74cab11424fa5bc15e4cba13157" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 42243 } } peers { permanent_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 33603 } } peers { permanent_uuid: "ca39e2b29dd34cc2a249e27cd9590c4e" member_type: NON_VOTER last_known_addr { host: "127.0.61.4" port: 39741 } attrs { promote: true } } } Master Summary UUID | Address | Status ----------------------------------+-------------------+--------- 490537774c7b411291a492c7d78633ca | 127.0.61.62:40441 | HEALTHY Flag | Value | Tags | Master ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+------------------+---------+---------- 0938c74cab11424fa5bc15e4cba13157 | 127.0.61.1:42243 | HEALTHY | 32da123829d846f2b8e23df043e04a15 | 127.0.61.3:44291 | HEALTHY | ca39e2b29dd34cc2a249e27cd9590c4e | 127.0.61.4:39741 | HEALTHY | f2a7d0ddeb8343839dd8dc3f3413a783 | 127.0.61.2:33603 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet 3485e0e47b994bf892b67638ce21a806 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 32da123829d846f2b8e23df043e04a15 (127.0.61.3:44291): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: INJECTED ERROR: tablet failed 0938c74cab11424fa5bc15e4cba13157 (127.0.61.1:42243): RUNNING f2a7d0ddeb8343839dd8dc3f3413a783 (127.0.61.2:33603): RUNNING [LEADER] All reported replicas are: A = 32da123829d846f2b8e23df043e04a15 B = 0938c74cab11424fa5bc15e4cba13157 C = f2a7d0ddeb8343839dd8dc3f3413a783 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | [config not available] | | | B | B C* | 1 | 2 | Yes C | B C* | 1 | 2 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 0 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+------------------+--------------- Small | ca39e2b29dd34cc2a249e27cd9590c4e | 127.0.61.4:39741 | 0 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED W0916 07:55:57.132266 2754 consensus_peers.cc:458] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 -> Peer ca39e2b29dd34cc2a249e27cd9590c4e (127.0.61.4:39741): Couldn't send request to peer ca39e2b29dd34cc2a249e27cd9590c4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3485e0e47b994bf892b67638ce21a806. This is attempt 1: this message will repeat every 5th retry. I0916 07:55:57.136708 244 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0916 07:55:57.168890 2622 catalog_manager.cc:3739] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 3485e0e47b994bf892b67638ce21a806 (attempt 2) I0916 07:55:57.651686 3130 ts_tablet_manager.cc:708] T 3485e0e47b994bf892b67638ce21a806 P ca39e2b29dd34cc2a249e27cd9590c4e: Initiating tablet copy from peer f2a7d0ddeb8343839dd8dc3f3413a783 (127.0.61.2:33603) I0916 07:55:57.652084 3130 tablet_copy_client.cc:241] T 3485e0e47b994bf892b67638ce21a806 P ca39e2b29dd34cc2a249e27cd9590c4e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.61.2:33603 I0916 07:55:57.661820 2801 tablet_copy_service.cc:135] P f2a7d0ddeb8343839dd8dc3f3413a783: Received BeginTabletCopySession request for tablet 3485e0e47b994bf892b67638ce21a806 from peer ca39e2b29dd34cc2a249e27cd9590c4e ({username='slave'} at 127.0.0.1:55256) I0916 07:55:57.662207 2801 tablet_copy_service.cc:156] P f2a7d0ddeb8343839dd8dc3f3413a783: Beginning new tablet copy session on tablet 3485e0e47b994bf892b67638ce21a806 from peer ca39e2b29dd34cc2a249e27cd9590c4e at {username='slave'} at 127.0.0.1:55256: session id = ca39e2b29dd34cc2a249e27cd9590c4e-3485e0e47b994bf892b67638ce21a806 W0916 07:55:57.915704 2723 consensus_meta.cc:220] T 3485e0e47b994bf892b67638ce21a806 P 0938c74cab11424fa5bc15e4cba13157: Time spent flushing consensus metadata: real 0.794s user 0.006s sys 0.000s W0916 07:55:57.915693 3091 consensus_meta.cc:220] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783: Time spent flushing consensus metadata: real 0.795s user 0.006s sys 0.000s I0916 07:55:57.920130 2622 catalog_manager.cc:3654] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3485e0e47b994bf892b67638ce21a806 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) W0916 07:55:57.920852 2622 catalog_manager.cc:3664] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3485e0e47b994bf892b67638ce21a806 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER failed with leader f2a7d0ddeb8343839dd8dc3f3413a783 (127.0.61.2:33603) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 2 but the committed config has opid_index of 3 I0916 07:55:57.923305 2638 catalog_manager.cc:4120] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 reported cstate change: config changed from index 2 to 3, NON_VOTER ca39e2b29dd34cc2a249e27cd9590c4e (127.0.61.4) added. New cstate: current_term: 1 leader_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0938c74cab11424fa5bc15e4cba13157" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 42243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 33603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca39e2b29dd34cc2a249e27cd9590c4e" member_type: NON_VOTER last_known_addr { host: "127.0.61.4" port: 39741 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0916 07:55:57.924007 2801 tablet_copy_source_session.cc:202] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783: Tablet Copy: opened 0 blocks and 1 log segments I0916 07:55:57.928772 3130 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 3485e0e47b994bf892b67638ce21a806. 1 dirs total, 0 dirs full, 0 dirs failed I0916 07:55:57.940655 3130 tablet_copy_client.cc:579] T 3485e0e47b994bf892b67638ce21a806 P ca39e2b29dd34cc2a249e27cd9590c4e: tablet copy: Starting download of 0 data blocks... I0916 07:55:57.941215 3130 tablet_copy_client.cc:542] T 3485e0e47b994bf892b67638ce21a806 P ca39e2b29dd34cc2a249e27cd9590c4e: tablet copy: Starting download of 1 WAL segments... I0916 07:55:57.943478 3130 tablet_copy_client.cc:418] T 3485e0e47b994bf892b67638ce21a806 P ca39e2b29dd34cc2a249e27cd9590c4e: tablet copy: Tablet Copy complete. Replacing tablet superblock. Master Summary UUID | Address | Status ----------------------------------+-------------------+--------- 490537774c7b411291a492c7d78633ca | 127.0.61.62:40441 | HEALTHY Flag | Value | Tags | Master ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+------------------+---------+---------- 0938c74cab11424fa5bc15e4cba13157 | 127.0.61.1:42243 | HEALTHY | 32da123829d846f2b8e23df043e04a15 | 127.0.61.3:44291 | HEALTHY | ca39e2b29dd34cc2a249e27cd9590c4e | 127.0.61.4:39741 | HEALTHY | f2a7d0ddeb8343839dd8dc3f3413a783 | 127.0.61.2:33603 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet 3485e0e47b994bf892b67638ce21a806 of table 'test-workload' is conflicted: 3 replicas' active configs disagree with the leader master's 0938c74cab11424fa5bc15e4cba13157 (127.0.61.1:42243): RUNNING f2a7d0ddeb8343839dd8dc3f3413a783 (127.0.61.2:33603): RUNNING [LEADER] All reported replicas are: A = 0938c74cab11424fa5bc15e4cba13157 B = f2a7d0ddeb8343839dd8dc3f3413a783 C = ca39e2b29dd34cc2a249e27cd9590c4e The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B* | | | Yes A | A B* C~ | 1 | 3 | Yes B | A B* C~ | 1 | 3 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 0 First Quartile | 0 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 1 Replicas | 2 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0916 07:55:57.970327 244 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors Master Summary UUID | Address | Status ----------------------------------+-------------------+--------- 490537774c7b411291a492c7d78633ca | 127.0.61.62:40441 | HEALTHY Flag | Value | Tags | Master ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+------------------+---------+---------- 0938c74cab11424fa5bc15e4cba13157 | 127.0.61.1:42243 | HEALTHY | 32da123829d846f2b8e23df043e04a15 | 127.0.61.3:44291 | HEALTHY | ca39e2b29dd34cc2a249e27cd9590c4e | 127.0.61.4:39741 | HEALTHY | f2a7d0ddeb8343839dd8dc3f3413a783 | 127.0.61.2:33603 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet 3485e0e47b994bf892b67638ce21a806 of table 'test-workload' is recovering: 1 on-going tablet copies 0938c74cab11424fa5bc15e4cba13157 (127.0.61.1:42243): RUNNING f2a7d0ddeb8343839dd8dc3f3413a783 (127.0.61.2:33603): RUNNING [LEADER] ca39e2b29dd34cc2a249e27cd9590c4e (127.0.61.4:39741): not running [NONVOTER] State: INITIALIZED Data state: TABLET_DATA_COPYING Last status: Tablet Copy: Replacing tablet superblock All reported replicas are: A = 0938c74cab11424fa5bc15e4cba13157 B = f2a7d0ddeb8343839dd8dc3f3413a783 C = ca39e2b29dd34cc2a249e27cd9590c4e The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B* C~ | | | Yes A | A B* C~ | 1 | 3 | Yes B | A B* C~ | 1 | 3 | Yes C | A B C~ | 1 | 3 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+------------+---------------+---------+------------+------------------+------------- test-workload | 3 | RECOVERING | 1 | 0 | 1 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 0 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+------------------+--------------- Small | 32da123829d846f2b8e23df043e04a15 | 127.0.61.3:44291 | 0 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0916 07:55:58.514264 244 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0916 07:55:59.015614 3098 ts_tablet_manager.cc:1453] T 3485e0e47b994bf892b67638ce21a806 P 32da123829d846f2b8e23df043e04a15: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1 I0916 07:55:59.015990 3098 log.cc:990] T 3485e0e47b994bf892b67638ce21a806 P 32da123829d846f2b8e23df043e04a15: Deleting WAL directory at /tmp/dist-test-taska4lOjz/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568620531643032-244/minicluster-data/ts-2-root/wals/3485e0e47b994bf892b67638ce21a806 I0916 07:55:59.016261 3130 tablet_bootstrap.cc:439] T 3485e0e47b994bf892b67638ce21a806 P ca39e2b29dd34cc2a249e27cd9590c4e: Bootstrap starting. I0916 07:55:59.038326 3130 tablet_bootstrap.cc:439] T 3485e0e47b994bf892b67638ce21a806 P ca39e2b29dd34cc2a249e27cd9590c4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0916 07:55:59.039108 3130 tablet_bootstrap.cc:439] T 3485e0e47b994bf892b67638ce21a806 P ca39e2b29dd34cc2a249e27cd9590c4e: Bootstrap complete. I0916 07:55:59.039587 3130 ts_tablet_manager.cc:1085] T 3485e0e47b994bf892b67638ce21a806 P ca39e2b29dd34cc2a249e27cd9590c4e: Time spent bootstrapping tablet: real 0.024s user 0.018s sys 0.000s I0916 07:55:59.041441 3130 raft_consensus.cc:340] T 3485e0e47b994bf892b67638ce21a806 P ca39e2b29dd34cc2a249e27cd9590c4e [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0938c74cab11424fa5bc15e4cba13157" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 42243 } } peers { permanent_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 33603 } } peers { permanent_uuid: "ca39e2b29dd34cc2a249e27cd9590c4e" member_type: NON_VOTER last_known_addr { host: "127.0.61.4" port: 39741 } attrs { promote: true } } I0916 07:55:59.041965 3130 raft_consensus.cc:705] T 3485e0e47b994bf892b67638ce21a806 P ca39e2b29dd34cc2a249e27cd9590c4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca39e2b29dd34cc2a249e27cd9590c4e, State: Initialized, Role: LEARNER I0916 07:55:59.042388 3130 consensus_queue.cc:229] T 3485e0e47b994bf892b67638ce21a806 P ca39e2b29dd34cc2a249e27cd9590c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0938c74cab11424fa5bc15e4cba13157" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 42243 } } peers { permanent_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 33603 } } peers { permanent_uuid: "ca39e2b29dd34cc2a249e27cd9590c4e" member_type: NON_VOTER last_known_addr { host: "127.0.61.4" port: 39741 } attrs { promote: true } } I0916 07:55:59.044507 3130 ts_tablet_manager.cc:1113] T 3485e0e47b994bf892b67638ce21a806 P ca39e2b29dd34cc2a249e27cd9590c4e: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s I0916 07:55:59.045581 2801 tablet_copy_service.cc:325] P f2a7d0ddeb8343839dd8dc3f3413a783: Request end of tablet copy session ca39e2b29dd34cc2a249e27cd9590c4e-3485e0e47b994bf892b67638ce21a806 received from {username='slave'} at 127.0.0.1:55256 I0916 07:55:59.045871 2801 tablet_copy_service.cc:413] P f2a7d0ddeb8343839dd8dc3f3413a783: ending tablet copy session ca39e2b29dd34cc2a249e27cd9590c4e-3485e0e47b994bf892b67638ce21a806 on tablet 3485e0e47b994bf892b67638ce21a806 with peer ca39e2b29dd34cc2a249e27cd9590c4e Master Summary UUID | Address | Status ----------------------------------+-------------------+--------- 490537774c7b411291a492c7d78633ca | 127.0.61.62:40441 | HEALTHY Flag | Value | Tags | Master ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+------------------+---------+---------- 0938c74cab11424fa5bc15e4cba13157 | 127.0.61.1:42243 | HEALTHY | 32da123829d846f2b8e23df043e04a15 | 127.0.61.3:44291 | HEALTHY | ca39e2b29dd34cc2a249e27cd9590c4e | 127.0.61.4:39741 | HEALTHY | f2a7d0ddeb8343839dd8dc3f3413a783 | 127.0.61.2:33603 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet 3485e0e47b994bf892b67638ce21a806 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's 0938c74cab11424fa5bc15e4cba13157 (127.0.61.1:42243): RUNNING f2a7d0ddeb8343839dd8dc3f3413a783 (127.0.61.2:33603): RUNNING [LEADER] ca39e2b29dd34cc2a249e27cd9590c4e (127.0.61.4:39741): RUNNING [NONVOTER] All reported replicas are: A = 0938c74cab11424fa5bc15e4cba13157 B = f2a7d0ddeb8343839dd8dc3f3413a783 C = ca39e2b29dd34cc2a249e27cd9590c4e The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B* C~ | | | Yes A | A B* C~ | 1 | 3 | Yes B | A B* C~ | 1 | 3 | Yes C | A B C~ | 1 | 3 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 0 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+------------------+--------------- Small | 32da123829d846f2b8e23df043e04a15 | 127.0.61.3:44291 | 0 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0916 07:55:59.205983 244 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0916 07:55:59.235780 2925 raft_consensus.cc:1126] T 3485e0e47b994bf892b67638ce21a806 P ca39e2b29dd34cc2a249e27cd9590c4e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[] I0916 07:55:59.681192 3200 raft_consensus.cc:974] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783: attempting to promote NON_VOTER ca39e2b29dd34cc2a249e27cd9590c4e to VOTER I0916 07:55:59.682755 3200 consensus_queue.cc:206] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0938c74cab11424fa5bc15e4cba13157" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 42243 } } peers { permanent_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 33603 } } peers { permanent_uuid: "ca39e2b29dd34cc2a249e27cd9590c4e" member_type: VOTER last_known_addr { host: "127.0.61.4" port: 39741 } attrs { promote: false } } I0916 07:55:59.687080 2925 raft_consensus.cc:1184] T 3485e0e47b994bf892b67638ce21a806 P ca39e2b29dd34cc2a249e27cd9590c4e [term 1 LEARNER]: Refusing update from remote peer f2a7d0ddeb8343839dd8dc3f3413a783: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I0916 07:55:59.687149 2723 raft_consensus.cc:1184] T 3485e0e47b994bf892b67638ce21a806 P 0938c74cab11424fa5bc15e4cba13157 [term 1 FOLLOWER]: Refusing update from remote peer f2a7d0ddeb8343839dd8dc3f3413a783: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I0916 07:55:59.688218 3199 consensus_queue.cc:984] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca39e2b29dd34cc2a249e27cd9590c4e" member_type: VOTER last_known_addr { host: "127.0.61.4" port: 39741 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I0916 07:55:59.689044 3167 consensus_queue.cc:984] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0938c74cab11424fa5bc15e4cba13157" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I0916 07:55:59.695341 3200 raft_consensus.cc:2792] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, ca39e2b29dd34cc2a249e27cd9590c4e (127.0.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0938c74cab11424fa5bc15e4cba13157" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 42243 } } peers { permanent_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 33603 } } peers { permanent_uuid: "ca39e2b29dd34cc2a249e27cd9590c4e" member_type: VOTER last_known_addr { host: "127.0.61.4" port: 39741 } attrs { promote: false } } } I0916 07:55:59.697033 2925 raft_consensus.cc:2792] T 3485e0e47b994bf892b67638ce21a806 P ca39e2b29dd34cc2a249e27cd9590c4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, ca39e2b29dd34cc2a249e27cd9590c4e (127.0.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0938c74cab11424fa5bc15e4cba13157" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 42243 } } peers { permanent_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 33603 } } peers { permanent_uuid: "ca39e2b29dd34cc2a249e27cd9590c4e" member_type: VOTER last_known_addr { host: "127.0.61.4" port: 39741 } attrs { promote: false } } } I0916 07:55:59.697064 2723 raft_consensus.cc:2792] T 3485e0e47b994bf892b67638ce21a806 P 0938c74cab11424fa5bc15e4cba13157 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, ca39e2b29dd34cc2a249e27cd9590c4e (127.0.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0938c74cab11424fa5bc15e4cba13157" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 42243 } } peers { permanent_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 33603 } } peers { permanent_uuid: "ca39e2b29dd34cc2a249e27cd9590c4e" member_type: VOTER last_known_addr { host: "127.0.61.4" port: 39741 } attrs { promote: false } } } I0916 07:55:59.699795 3098 rpcz_store.cc:269] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58446 (request call id 2) took 2612ms. Request Metrics: {"tablet-delete.queue_time_us":8371,"thread_start_us":6014,"threads_started":1} I0916 07:55:59.701576 2621 catalog_manager.cc:3490] TS 32da123829d846f2b8e23df043e04a15 (127.0.61.3:44291): tablet 3485e0e47b994bf892b67638ce21a806 (table test-workload [id=81c4fe0d20654d429dedb2035fe0c0a4]) successfully deleted I0916 07:55:59.818305 2638 catalog_manager.cc:4120] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 reported cstate change: config changed from index 3 to 4, ca39e2b29dd34cc2a249e27cd9590c4e (127.0.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0938c74cab11424fa5bc15e4cba13157" member_type: VOTER last_known_addr { host: "127.0.61.1" port: 42243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2a7d0ddeb8343839dd8dc3f3413a783" member_type: VOTER last_known_addr { host: "127.0.61.2" port: 33603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca39e2b29dd34cc2a249e27cd9590c4e" member_type: VOTER last_known_addr { host: "127.0.61.4" port: 39741 } attrs { promote: false } health_report { overall_health: HEALTHY } } } Master Summary UUID | Address | Status ----------------------------------+-------------------+--------- 490537774c7b411291a492c7d78633ca | 127.0.61.62:40441 | HEALTHY Flag | Value | Tags | Master ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+------------------+---------+---------- 0938c74cab11424fa5bc15e4cba13157 | 127.0.61.1:42243 | HEALTHY | 32da123829d846f2b8e23df043e04a15 | 127.0.61.3:44291 | HEALTHY | ca39e2b29dd34cc2a249e27cd9590c4e | 127.0.61.4:39741 | HEALTHY | f2a7d0ddeb8343839dd8dc3f3413a783 | 127.0.61.2:33603 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 5 server(s) checked Tablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+---------+---------------+---------+------------+------------------+------------- test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 0 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+------------------+--------------- Small | 32da123829d846f2b8e23df043e04a15 | 127.0.61.3:44291 | 0 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0916 07:56:00.154894 244 log_verifier.cc:121] Checking tablet 3485e0e47b994bf892b67638ce21a806 I0916 07:56:00.224716 244 log_verifier.cc:172] Verified matching terms for 4 ops in tablet 3485e0e47b994bf892b67638ce21a806 I0916 07:56:00.229568 244 tablet_server.cc:159] TabletServer@127.0.61.1:42243 shutting down... I0916 07:56:00.250730 244 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0916 07:56:00.251008 244 tablet_replica.cc:273] T 3485e0e47b994bf892b67638ce21a806 P 0938c74cab11424fa5bc15e4cba13157: stopping tablet replica I0916 07:56:00.251315 244 raft_consensus.cc:2147] T 3485e0e47b994bf892b67638ce21a806 P 0938c74cab11424fa5bc15e4cba13157 [term 1 FOLLOWER]: Raft consensus shutting down. I0916 07:56:00.251569 244 raft_consensus.cc:2174] T 3485e0e47b994bf892b67638ce21a806 P 0938c74cab11424fa5bc15e4cba13157 [term 1 FOLLOWER]: Raft consensus is shut down! I0916 07:56:00.270902 244 tablet_server.cc:173] TabletServer@127.0.61.1:42243 shutdown complete. I0916 07:56:00.278235 244 tablet_server.cc:159] TabletServer@127.0.61.2:33603 shutting down... I0916 07:56:00.295133 244 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0916 07:56:00.295433 244 tablet_replica.cc:273] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783: stopping tablet replica I0916 07:56:00.295794 244 raft_consensus.cc:2147] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 [term 1 LEADER]: Raft consensus shutting down. I0916 07:56:00.296365 244 raft_consensus.cc:2174] T 3485e0e47b994bf892b67638ce21a806 P f2a7d0ddeb8343839dd8dc3f3413a783 [term 1 FOLLOWER]: Raft consensus is shut down! I0916 07:56:00.315829 244 tablet_server.cc:173] TabletServer@127.0.61.2:33603 shutdown complete. I0916 07:56:00.323841 244 tablet_server.cc:159] TabletServer@127.0.61.3:44291 shutting down... I0916 07:56:00.342787 244 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0916 07:56:00.359118 244 tablet_server.cc:173] TabletServer@127.0.61.3:44291 shutdown complete. I0916 07:56:00.366257 244 tablet_server.cc:159] TabletServer@127.0.61.4:39741 shutting down... I0916 07:56:00.382242 244 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0916 07:56:00.382531 244 tablet_replica.cc:273] T 3485e0e47b994bf892b67638ce21a806 P ca39e2b29dd34cc2a249e27cd9590c4e: stopping tablet replica I0916 07:56:00.382817 244 raft_consensus.cc:2147] T 3485e0e47b994bf892b67638ce21a806 P ca39e2b29dd34cc2a249e27cd9590c4e [term 1 FOLLOWER]: Raft consensus shutting down. I0916 07:56:00.383054 244 raft_consensus.cc:2174] T 3485e0e47b994bf892b67638ce21a806 P ca39e2b29dd34cc2a249e27cd9590c4e [term 1 FOLLOWER]: Raft consensus is shut down! I0916 07:56:00.401983 244 tablet_server.cc:173] TabletServer@127.0.61.4:39741 shutdown complete. I0916 07:56:00.408959 244 master.cc:247] Master@127.0.61.62:40441 shutting down... I0916 07:56:00.423669 244 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 490537774c7b411291a492c7d78633ca [term 1 LEADER]: Raft consensus shutting down. I0916 07:56:00.423986 244 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 490537774c7b411291a492c7d78633ca [term 1 FOLLOWER]: Raft consensus is shut down! I0916 07:56:00.424160 244 tablet_replica.cc:273] T 00000000000000000000000000000000 P 490537774c7b411291a492c7d78633ca: stopping tablet replica I0916 07:56:00.441747 244 master.cc:258] Master@127.0.61.62:40441 shutdown complete. [ OK ] FailedTabletsAreReplacedITest.OneReplica/1 (6344 ms) [----------] 2 tests from FailedTabletsAreReplacedITest (13402 ms total) [----------] Global test environment tear-down [==========] 6 tests from 2 test cases ran. (28725 ms total) [ PASSED ] 6 tests. ThreadSanitizer: reported 2 warnings