[==========] Running 6 tests from 2 test cases. [----------] Global test environment set-up. [----------] 4 tests from TsTabletManagerITest [ RUN ] TsTabletManagerITest.TestReportNewLeaderOnLeaderChange WARNING: Logging before InitGoogleLogging() is written to STDERR W0913 23:13:38.754995 13524 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.851s user 0.002s sys 0.000s W0913 23:13:38.755559 13524 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.851s user 0.003s sys 0.000s I0913 23:13:38.758499 13524 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.13.53.62:46449 I0913 23:13:38.786720 13524 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 14200us I0913 23:13:38.787544 13524 fs_manager.cc:263] Metadata directory not provided I0913 23:13:38.787770 13524 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0913 23:13:38.788213 13524 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0913 23:13:38.788439 13524 server_base.cc:449] Attempting to create new FS layout instead I0913 23:13:38.798344 13524 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568416417781642-13524/minicluster-data/master-0-root/instance: uuid: "054e615467164e24aae07e8737be67e6" format_stamp: "Formatted at 2019-09-13 23:13:38 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:38.815141 13524 fs_manager.cc:503] Time spent creating directory manager: real 0.016s user 0.015s sys 0.001s I0913 23:13:38.816437 13524 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0913 23:13:38.817039 13524 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0913 23:13:38.825703 13524 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s I0913 23:13:38.825867 13524 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568416417781642-13524/minicluster-data/master-0-root uuid: "054e615467164e24aae07e8737be67e6" format_stamp: "Formatted at 2019-09-13 23:13:38 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:38.826251 13524 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568416417781642-13524/minicluster-data/master-0-root metadata directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568416417781642-13524/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568416417781642-13524/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0913 23:13:38.851104 13524 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0913 23:13:38.913244 13524 rpc_server.cc:205] RPC server started. Bound to: 127.13.53.62:46449 I0913 23:13:38.916662 13524 webserver.cc:374] Webserver started at http://127.13.53.62:46757/ using document root and password file I0913 23:13:38.920357 13589 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0913 23:13:38.938194 13589 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 054e615467164e24aae07e8737be67e6: Bootstrap starting. I0913 23:13:38.941283 13589 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 054e615467164e24aae07e8737be67e6: No blocks or log segments found. Creating new log. I0913 23:13:38.942776 13589 log.cc:525] T 00000000000000000000000000000000 P 054e615467164e24aae07e8737be67e6: Log is configured to *not* fsync() on all Append() calls I0913 23:13:38.945996 13589 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 054e615467164e24aae07e8737be67e6: No bootstrap required, opened a new log I0913 23:13:38.961889 13589 raft_consensus.cc:340] T 00000000000000000000000000000000 P 054e615467164e24aae07e8737be67e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054e615467164e24aae07e8737be67e6" member_type: VOTER } I0913 23:13:38.962357 13589 raft_consensus.cc:366] T 00000000000000000000000000000000 P 054e615467164e24aae07e8737be67e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0913 23:13:38.962524 13589 raft_consensus.cc:705] T 00000000000000000000000000000000 P 054e615467164e24aae07e8737be67e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 054e615467164e24aae07e8737be67e6, State: Initialized, Role: FOLLOWER I0913 23:13:38.963110 13589 consensus_queue.cc:229] T 00000000000000000000000000000000 P 054e615467164e24aae07e8737be67e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054e615467164e24aae07e8737be67e6" member_type: VOTER } I0913 23:13:38.963516 13589 raft_consensus.cc:380] T 00000000000000000000000000000000 P 054e615467164e24aae07e8737be67e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0913 23:13:38.963691 13589 raft_consensus.cc:472] T 00000000000000000000000000000000 P 054e615467164e24aae07e8737be67e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0913 23:13:38.963910 13589 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 054e615467164e24aae07e8737be67e6 [term 0 FOLLOWER]: Advancing to term 1 I0913 23:13:38.967113 13589 raft_consensus.cc:494] T 00000000000000000000000000000000 P 054e615467164e24aae07e8737be67e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054e615467164e24aae07e8737be67e6" member_type: VOTER } I0913 23:13:38.967694 13589 leader_election.cc:310] T 00000000000000000000000000000000 P 054e615467164e24aae07e8737be67e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 054e615467164e24aae07e8737be67e6; no voters: I0913 23:13:38.969543 13591 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 054e615467164e24aae07e8737be67e6 [term 1 FOLLOWER]: Leader election won for term 1 I0913 23:13:38.970786 13589 leader_election.cc:296] T 00000000000000000000000000000000 P 054e615467164e24aae07e8737be67e6 [CANDIDATE]: Term 1 election: Requested vote from peers I0913 23:13:39.070022 13591 raft_consensus.cc:667] T 00000000000000000000000000000000 P 054e615467164e24aae07e8737be67e6 [term 1 LEADER]: Becoming Leader. State: Replica: 054e615467164e24aae07e8737be67e6, State: Running, Role: LEADER I0913 23:13:39.070684 13591 consensus_queue.cc:206] T 00000000000000000000000000000000 P 054e615467164e24aae07e8737be67e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054e615467164e24aae07e8737be67e6" member_type: VOTER } I0913 23:13:39.075088 13593 sys_catalog.cc:340] T 00000000000000000000000000000000 P 054e615467164e24aae07e8737be67e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "054e615467164e24aae07e8737be67e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054e615467164e24aae07e8737be67e6" member_type: VOTER } } I0913 23:13:39.075732 13593 sys_catalog.cc:343] T 00000000000000000000000000000000 P 054e615467164e24aae07e8737be67e6 [sys.catalog]: This master's current role is: LEADER I0913 23:13:39.075832 13592 sys_catalog.cc:340] T 00000000000000000000000000000000 P 054e615467164e24aae07e8737be67e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 054e615467164e24aae07e8737be67e6. Latest consensus state: current_term: 1 leader_uuid: "054e615467164e24aae07e8737be67e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054e615467164e24aae07e8737be67e6" member_type: VOTER } } I0913 23:13:39.076236 13589 sys_catalog.cc:340] T 00000000000000000000000000000000 P 054e615467164e24aae07e8737be67e6 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "054e615467164e24aae07e8737be67e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054e615467164e24aae07e8737be67e6" member_type: VOTER } } I0913 23:13:39.076529 13592 sys_catalog.cc:343] T 00000000000000000000000000000000 P 054e615467164e24aae07e8737be67e6 [sys.catalog]: This master's current role is: LEADER I0913 23:13:39.076719 13589 sys_catalog.cc:343] T 00000000000000000000000000000000 P 054e615467164e24aae07e8737be67e6 [sys.catalog]: This master's current role is: LEADER I0913 23:13:39.080605 13589 sys_catalog.cc:428] T 00000000000000000000000000000000 P 054e615467164e24aae07e8737be67e6 [sys.catalog]: configured and running, proceeding with master startup. I0913 23:13:39.082242 13595 catalog_manager.cc:1071] Loading table and tablet metadata into memory... I0913 23:13:39.082640 13524 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0913 23:13:39.087730 13595 catalog_manager.cc:1082] Initializing Kudu internal certificate authority... I0913 23:13:39.144721 13595 catalog_manager.cc:966] Generated new certificate authority record I0913 23:13:39.146430 13595 catalog_manager.cc:1091] Loading token signing keys... I0913 23:13:39.157696 13595 catalog_manager.cc:4394] T 00000000000000000000000000000000 P 054e615467164e24aae07e8737be67e6: Generated new TSK 0 I0913 23:13:39.235643 13524 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 14700us I0913 23:13:39.236119 13524 fs_manager.cc:263] Metadata directory not provided I0913 23:13:39.236320 13524 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0913 23:13:39.236591 13524 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0913 23:13:39.236739 13524 server_base.cc:449] Attempting to create new FS layout instead I0913 23:13:39.237995 13524 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568416417781642-13524/minicluster-data/ts-0-root/instance: uuid: "5377d9dbe0a94202a7ba43eecf52397f" format_stamp: "Formatted at 2019-09-13 23:13:39 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:39.255771 13524 fs_manager.cc:503] Time spent creating directory manager: real 0.017s user 0.012s sys 0.004s I0913 23:13:39.257882 13524 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0913 23:13:39.271735 13524 fs_manager.cc:419] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s I0913 23:13:39.271961 13524 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568416417781642-13524/minicluster-data/ts-0-root uuid: "5377d9dbe0a94202a7ba43eecf52397f" format_stamp: "Formatted at 2019-09-13 23:13:39 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:39.272385 13524 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568416417781642-13524/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568416417781642-13524/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568416417781642-13524/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0913 23:13:39.349825 13524 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0913 23:13:39.350028 13524 ts_tablet_manager.cc:403] Registered 0 tablets I0913 23:13:39.490234 13524 rpc_server.cc:205] RPC server started. Bound to: 127.13.53.1:41723 I0913 23:13:39.492671 13524 webserver.cc:374] Webserver started at http://127.13.53.1:35821/ using document root and password file I0913 23:13:39.516067 13664 heartbeater.cc:322] Connected to a master server at 127.13.53.62:46449 I0913 23:13:39.516443 13664 heartbeater.cc:406] Registering TS with master... I0913 23:13:39.517026 13664 heartbeater.cc:455] Master 127.13.53.62:46449 requested a full tablet report, sending... I0913 23:13:39.518196 13524 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 14700us I0913 23:13:39.518631 13524 fs_manager.cc:263] Metadata directory not provided I0913 23:13:39.518817 13524 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0913 23:13:39.519058 13524 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0913 23:13:39.519210 13524 server_base.cc:449] Attempting to create new FS layout instead I0913 23:13:39.519457 13556 ts_manager.cc:161] Registered new tserver with Master: 5377d9dbe0a94202a7ba43eecf52397f (127.13.53.1:41723) I0913 23:13:39.520401 13524 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568416417781642-13524/minicluster-data/ts-1-root/instance: uuid: "d215c4eb31bd4e1c84973b81be2c2705" format_stamp: "Formatted at 2019-09-13 23:13:39 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:39.521486 13556 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44760 I0913 23:13:39.534555 13524 fs_manager.cc:503] Time spent creating directory manager: real 0.014s user 0.011s sys 0.004s I0913 23:13:39.536917 13524 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0913 23:13:39.549109 13524 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0913 23:13:39.549329 13524 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568416417781642-13524/minicluster-data/ts-1-root uuid: "d215c4eb31bd4e1c84973b81be2c2705" format_stamp: "Formatted at 2019-09-13 23:13:39 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:39.549630 13524 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568416417781642-13524/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568416417781642-13524/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568416417781642-13524/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0913 23:13:39.582633 13524 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0913 23:13:39.582852 13524 ts_tablet_manager.cc:403] Registered 0 tablets I0913 23:13:39.663326 13524 rpc_server.cc:205] RPC server started. Bound to: 127.13.53.2:43915 I0913 23:13:39.665628 13524 webserver.cc:374] Webserver started at http://127.13.53.2:44577/ using document root and password file I0913 23:13:39.681360 13732 heartbeater.cc:322] Connected to a master server at 127.13.53.62:46449 I0913 23:13:39.681658 13732 heartbeater.cc:406] Registering TS with master... I0913 23:13:39.682169 13732 heartbeater.cc:455] Master 127.13.53.62:46449 requested a full tablet report, sending... I0913 23:13:39.683692 13556 ts_manager.cc:161] Registered new tserver with Master: d215c4eb31bd4e1c84973b81be2c2705 (127.13.53.2:43915) I0913 23:13:39.683846 13524 internal_mini_cluster.cc:328] 2 TS(s) registered with all masters after 0.012361938s I0913 23:13:39.685484 13556 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44762 I0913 23:13:39.719858 13556 catalog_manager.cc:1425] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44764: name: "test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 2 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0913 23:13:39.738253 13556 catalog_manager.cc:1590] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers. I0913 23:13:39.794706 13699 tablet_service.cc:1022] Processing CreateTablet for tablet 1cf446921adc45968995f85692029a33 (table=test-table [id=f82647784dd54b2e8cf65fbce2f24f2e]), partition=RANGE (key) PARTITION UNBOUNDED I0913 23:13:39.795943 13699 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 1cf446921adc45968995f85692029a33. 1 dirs total, 0 dirs full, 0 dirs failed I0913 23:13:39.804098 13631 tablet_service.cc:1022] Processing CreateTablet for tablet 1cf446921adc45968995f85692029a33 (table=test-table [id=f82647784dd54b2e8cf65fbce2f24f2e]), partition=RANGE (key) PARTITION UNBOUNDED I0913 23:13:39.805011 13631 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 1cf446921adc45968995f85692029a33. 1 dirs total, 0 dirs full, 0 dirs failed I0913 23:13:39.807983 13748 tablet_bootstrap.cc:439] T 1cf446921adc45968995f85692029a33 P d215c4eb31bd4e1c84973b81be2c2705: Bootstrap starting. I0913 23:13:39.811782 13748 tablet_bootstrap.cc:590] T 1cf446921adc45968995f85692029a33 P d215c4eb31bd4e1c84973b81be2c2705: No blocks or log segments found. Creating new log. I0913 23:13:39.816839 13748 tablet_bootstrap.cc:439] T 1cf446921adc45968995f85692029a33 P d215c4eb31bd4e1c84973b81be2c2705: No bootstrap required, opened a new log I0913 23:13:39.817385 13748 ts_tablet_manager.cc:1085] T 1cf446921adc45968995f85692029a33 P d215c4eb31bd4e1c84973b81be2c2705: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s I0913 23:13:39.819710 13748 raft_consensus.cc:340] T 1cf446921adc45968995f85692029a33 P d215c4eb31bd4e1c84973b81be2c2705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5377d9dbe0a94202a7ba43eecf52397f" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 41723 } } peers { permanent_uuid: "d215c4eb31bd4e1c84973b81be2c2705" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 43915 } } I0913 23:13:39.820588 13750 tablet_bootstrap.cc:439] T 1cf446921adc45968995f85692029a33 P 5377d9dbe0a94202a7ba43eecf52397f: Bootstrap starting. I0913 23:13:39.820585 13748 raft_consensus.cc:705] T 1cf446921adc45968995f85692029a33 P d215c4eb31bd4e1c84973b81be2c2705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d215c4eb31bd4e1c84973b81be2c2705, State: Initialized, Role: FOLLOWER I0913 23:13:39.821223 13748 consensus_queue.cc:229] T 1cf446921adc45968995f85692029a33 P d215c4eb31bd4e1c84973b81be2c2705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5377d9dbe0a94202a7ba43eecf52397f" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 41723 } } peers { permanent_uuid: "d215c4eb31bd4e1c84973b81be2c2705" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 43915 } } I0913 23:13:39.824487 13732 heartbeater.cc:447] Master 127.13.53.62:46449 was elected leader, sending a full tablet report... I0913 23:13:39.825084 13750 tablet_bootstrap.cc:590] T 1cf446921adc45968995f85692029a33 P 5377d9dbe0a94202a7ba43eecf52397f: No blocks or log segments found. Creating new log. I0913 23:13:39.826244 13748 ts_tablet_manager.cc:1113] T 1cf446921adc45968995f85692029a33 P d215c4eb31bd4e1c84973b81be2c2705: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s I0913 23:13:39.838572 13750 tablet_bootstrap.cc:439] T 1cf446921adc45968995f85692029a33 P 5377d9dbe0a94202a7ba43eecf52397f: No bootstrap required, opened a new log I0913 23:13:39.838973 13750 ts_tablet_manager.cc:1085] T 1cf446921adc45968995f85692029a33 P 5377d9dbe0a94202a7ba43eecf52397f: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.005s I0913 23:13:39.840440 13750 raft_consensus.cc:340] T 1cf446921adc45968995f85692029a33 P 5377d9dbe0a94202a7ba43eecf52397f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5377d9dbe0a94202a7ba43eecf52397f" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 41723 } } peers { permanent_uuid: "d215c4eb31bd4e1c84973b81be2c2705" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 43915 } } I0913 23:13:39.841013 13750 raft_consensus.cc:705] T 1cf446921adc45968995f85692029a33 P 5377d9dbe0a94202a7ba43eecf52397f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5377d9dbe0a94202a7ba43eecf52397f, State: Initialized, Role: FOLLOWER I0913 23:13:39.841346 13750 consensus_queue.cc:229] T 1cf446921adc45968995f85692029a33 P 5377d9dbe0a94202a7ba43eecf52397f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5377d9dbe0a94202a7ba43eecf52397f" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 41723 } } peers { permanent_uuid: "d215c4eb31bd4e1c84973b81be2c2705" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 43915 } } I0913 23:13:39.843621 13664 heartbeater.cc:447] Master 127.13.53.62:46449 was elected leader, sending a full tablet report... I0913 23:13:39.843850 13750 ts_tablet_manager.cc:1113] T 1cf446921adc45968995f85692029a33 P 5377d9dbe0a94202a7ba43eecf52397f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s I0913 23:13:39.866966 13524 ts_tablet_manager-itest.cc:324] Electing peer 0... I0913 23:13:39.867175 13524 raft_consensus.cc:405] T 1cf446921adc45968995f85692029a33 P 5377d9dbe0a94202a7ba43eecf52397f [term 0 FOLLOWER]: Emulating election... I0913 23:13:39.867381 13524 raft_consensus.cc:2890] T 1cf446921adc45968995f85692029a33 P 5377d9dbe0a94202a7ba43eecf52397f [term 0 FOLLOWER]: Advancing to term 1 I0913 23:13:39.871057 13524 raft_consensus.cc:667] T 1cf446921adc45968995f85692029a33 P 5377d9dbe0a94202a7ba43eecf52397f [term 1 LEADER]: Becoming Leader. State: Replica: 5377d9dbe0a94202a7ba43eecf52397f, State: Running, Role: LEADER W0913 23:13:39.871507 13664 heartbeater.cc:571] Failed to heartbeat to 127.13.53.62:46449 (0 consecutive failures): IO error: failing all heartbeats for tests I0913 23:13:39.871557 13524 consensus_queue.cc:206] T 1cf446921adc45968995f85692029a33 P 5377d9dbe0a94202a7ba43eecf52397f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5377d9dbe0a94202a7ba43eecf52397f" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 41723 } } peers { permanent_uuid: "d215c4eb31bd4e1c84973b81be2c2705" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 43915 } } I0913 23:13:39.875128 13524 ts_tablet_manager-itest.cc:327] Waiting for servers to agree... I0913 23:13:39.896453 13524 cluster_itest_util.cc:259] Not converged past 1 yet: 1.1 0.0 W0913 23:13:39.922448 13733 tablet.cc:1762] T 1cf446921adc45968995f85692029a33 P d215c4eb31bd4e1c84973b81be2c2705: Can't schedule compaction. Clean time has not been advanced past its initial value. I0913 23:13:40.000432 13524 cluster_itest_util.cc:259] Not converged past 1 yet: 1.1 0.0 I0913 23:13:40.204047 13524 cluster_itest_util.cc:259] Not converged past 1 yet: 1.1 0.0 I0913 23:13:40.430233 13709 raft_consensus.cc:2890] T 1cf446921adc45968995f85692029a33 P d215c4eb31bd4e1c84973b81be2c2705 [term 0 FOLLOWER]: Advancing to term 1 I0913 23:13:40.434948 13756 consensus_queue.cc:984] T 1cf446921adc45968995f85692029a33 P 5377d9dbe0a94202a7ba43eecf52397f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d215c4eb31bd4e1c84973b81be2c2705" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 43915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0913 23:13:40.508771 13524 ts_tablet_manager-itest.cc:324] Electing peer 1... I0913 23:13:40.508941 13524 raft_consensus.cc:405] T 1cf446921adc45968995f85692029a33 P d215c4eb31bd4e1c84973b81be2c2705 [term 1 FOLLOWER]: Emulating election... I0913 23:13:40.509121 13524 raft_consensus.cc:2890] T 1cf446921adc45968995f85692029a33 P d215c4eb31bd4e1c84973b81be2c2705 [term 1 FOLLOWER]: Advancing to term 2 I0913 23:13:40.512538 13524 raft_consensus.cc:667] T 1cf446921adc45968995f85692029a33 P d215c4eb31bd4e1c84973b81be2c2705 [term 2 LEADER]: Becoming Leader. State: Replica: d215c4eb31bd4e1c84973b81be2c2705, State: Running, Role: LEADER I0913 23:13:40.512953 13524 consensus_queue.cc:206] T 1cf446921adc45968995f85692029a33 P d215c4eb31bd4e1c84973b81be2c2705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5377d9dbe0a94202a7ba43eecf52397f" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 41723 } } peers { permanent_uuid: "d215c4eb31bd4e1c84973b81be2c2705" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 43915 } } I0913 23:13:40.514281 13524 ts_tablet_manager-itest.cc:327] Waiting for servers to agree... I0913 23:13:40.517056 13524 cluster_itest_util.cc:259] Not converged past 2 yet: 1.1 2.2 I0913 23:13:40.620498 13524 cluster_itest_util.cc:259] Not converged past 2 yet: 1.1 2.2 I0913 23:13:40.823879 13524 cluster_itest_util.cc:259] Not converged past 2 yet: 1.1 2.2 I0913 23:13:40.848071 13709 raft_consensus.cc:1149] T 1cf446921adc45968995f85692029a33 P d215c4eb31bd4e1c84973b81be2c2705 [term 2 LEADER]: Rejecting Update request from peer 5377d9dbe0a94202a7ba43eecf52397f for earlier term 1. Current term is 2. Ops: [] I0913 23:13:40.849157 13764 consensus_queue.cc:995] T 1cf446921adc45968995f85692029a33 P 5377d9dbe0a94202a7ba43eecf52397f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d215c4eb31bd4e1c84973b81be2c2705" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 43915 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0913 23:13:40.849683 13764 consensus_queue.cc:1162] T 1cf446921adc45968995f85692029a33 P 5377d9dbe0a94202a7ba43eecf52397f [LEADER]: Peer d215c4eb31bd4e1c84973b81be2c2705 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1 I0913 23:13:40.849992 13761 raft_consensus.cc:2885] T 1cf446921adc45968995f85692029a33 P 5377d9dbe0a94202a7ba43eecf52397f [term 1 LEADER]: Stepping down as leader of term 1 I0913 23:13:40.850247 13761 raft_consensus.cc:705] T 1cf446921adc45968995f85692029a33 P 5377d9dbe0a94202a7ba43eecf52397f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5377d9dbe0a94202a7ba43eecf52397f, State: Running, Role: LEADER I0913 23:13:40.850554 13761 consensus_queue.cc:229] T 1cf446921adc45968995f85692029a33 P 5377d9dbe0a94202a7ba43eecf52397f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5377d9dbe0a94202a7ba43eecf52397f" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 41723 } } peers { permanent_uuid: "d215c4eb31bd4e1c84973b81be2c2705" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 43915 } } I0913 23:13:40.851289 13761 raft_consensus.cc:2890] T 1cf446921adc45968995f85692029a33 P 5377d9dbe0a94202a7ba43eecf52397f [term 1 FOLLOWER]: Advancing to term 2 I0913 23:13:40.976162 13641 raft_consensus.cc:1184] T 1cf446921adc45968995f85692029a33 P 5377d9dbe0a94202a7ba43eecf52397f [term 2 FOLLOWER]: Refusing update from remote peer d215c4eb31bd4e1c84973b81be2c2705: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch) I0913 23:13:40.977279 13759 consensus_queue.cc:984] T 1cf446921adc45968995f85692029a33 P d215c4eb31bd4e1c84973b81be2c2705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5377d9dbe0a94202a7ba43eecf52397f" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 41723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0913 23:13:41.128537 13524 ts_tablet_manager-itest.cc:324] Electing peer 1... I0913 23:13:41.128767 13524 raft_consensus.cc:405] T 1cf446921adc45968995f85692029a33 P d215c4eb31bd4e1c84973b81be2c2705 [term 2 LEADER]: Emulating election... I0913 23:13:41.128935 13524 raft_consensus.cc:2885] T 1cf446921adc45968995f85692029a33 P d215c4eb31bd4e1c84973b81be2c2705 [term 2 LEADER]: Stepping down as leader of term 2 I0913 23:13:41.129053 13524 raft_consensus.cc:705] T 1cf446921adc45968995f85692029a33 P d215c4eb31bd4e1c84973b81be2c2705 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d215c4eb31bd4e1c84973b81be2c2705, State: Running, Role: LEADER I0913 23:13:41.129325 13524 consensus_queue.cc:229] T 1cf446921adc45968995f85692029a33 P d215c4eb31bd4e1c84973b81be2c2705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5377d9dbe0a94202a7ba43eecf52397f" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 41723 } } peers { permanent_uuid: "d215c4eb31bd4e1c84973b81be2c2705" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 43915 } } I0913 23:13:41.129865 13524 raft_consensus.cc:2890] T 1cf446921adc45968995f85692029a33 P d215c4eb31bd4e1c84973b81be2c2705 [term 2 FOLLOWER]: Advancing to term 3 I0913 23:13:41.133249 13524 raft_consensus.cc:667] T 1cf446921adc45968995f85692029a33 P d215c4eb31bd4e1c84973b81be2c2705 [term 3 LEADER]: Becoming Leader. State: Replica: d215c4eb31bd4e1c84973b81be2c2705, State: Running, Role: LEADER I0913 23:13:41.133632 13524 consensus_queue.cc:206] T 1cf446921adc45968995f85692029a33 P d215c4eb31bd4e1c84973b81be2c2705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5377d9dbe0a94202a7ba43eecf52397f" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 41723 } } peers { permanent_uuid: "d215c4eb31bd4e1c84973b81be2c2705" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 43915 } } I0913 23:13:41.135020 13524 ts_tablet_manager-itest.cc:327] Waiting for servers to agree... I0913 23:13:41.138226 13524 cluster_itest_util.cc:259] Not converged past 3 yet: 2.2 3.3 I0913 23:13:41.241641 13524 cluster_itest_util.cc:259] Not converged past 3 yet: 2.2 3.3 I0913 23:13:41.445075 13524 cluster_itest_util.cc:259] Not converged past 3 yet: 2.2 3.3 I0913 23:13:41.572232 13641 raft_consensus.cc:2890] T 1cf446921adc45968995f85692029a33 P 5377d9dbe0a94202a7ba43eecf52397f [term 2 FOLLOWER]: Advancing to term 3 I0913 23:13:41.576256 13641 raft_consensus.cc:1184] T 1cf446921adc45968995f85692029a33 P 5377d9dbe0a94202a7ba43eecf52397f [term 3 FOLLOWER]: Refusing update from remote peer d215c4eb31bd4e1c84973b81be2c2705: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch) I0913 23:13:41.577294 13759 consensus_queue.cc:984] T 1cf446921adc45968995f85692029a33 P d215c4eb31bd4e1c84973b81be2c2705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5377d9dbe0a94202a7ba43eecf52397f" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 41723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0913 23:13:41.750516 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:13:41.769585 13524 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0913 23:13:41.769865 13524 tablet_replica.cc:273] T 1cf446921adc45968995f85692029a33 P 5377d9dbe0a94202a7ba43eecf52397f: stopping tablet replica I0913 23:13:41.770262 13524 raft_consensus.cc:2147] T 1cf446921adc45968995f85692029a33 P 5377d9dbe0a94202a7ba43eecf52397f [term 3 FOLLOWER]: Raft consensus shutting down. I0913 23:13:41.770552 13524 raft_consensus.cc:2174] T 1cf446921adc45968995f85692029a33 P 5377d9dbe0a94202a7ba43eecf52397f [term 3 FOLLOWER]: Raft consensus is shut down! I0913 23:13:41.847349 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:13:41.847520 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:13:41.847841 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:13:41.853674 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:13:41.867766 13524 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0913 23:13:41.868026 13524 tablet_replica.cc:273] T 1cf446921adc45968995f85692029a33 P d215c4eb31bd4e1c84973b81be2c2705: stopping tablet replica I0913 23:13:41.868301 13524 raft_consensus.cc:2147] T 1cf446921adc45968995f85692029a33 P d215c4eb31bd4e1c84973b81be2c2705 [term 3 LEADER]: Raft consensus shutting down. I0913 23:13:41.868717 13524 raft_consensus.cc:2174] T 1cf446921adc45968995f85692029a33 P d215c4eb31bd4e1c84973b81be2c2705 [term 3 FOLLOWER]: Raft consensus is shut down! I0913 23:13:41.886304 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:13:41.886461 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:13:41.886742 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:13:41.891917 13524 master.cc:247] Master@127.13.53.62:46449 shutting down... I0913 23:13:41.947819 13524 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 054e615467164e24aae07e8737be67e6 [term 1 LEADER]: Raft consensus shutting down. I0913 23:13:41.948174 13524 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 054e615467164e24aae07e8737be67e6 [term 1 FOLLOWER]: Raft consensus is shut down! I0913 23:13:41.948405 13524 tablet_replica.cc:273] T 00000000000000000000000000000000 P 054e615467164e24aae07e8737be67e6: stopping tablet replica I0913 23:13:41.964323 13524 master.cc:258] Master@127.13.53.62:46449 shutdown complete. [ OK ] TsTabletManagerITest.TestReportNewLeaderOnLeaderChange (4116 ms) [ RUN ] TsTabletManagerITest.ReportOnReplicaHealthStatus I0913 23:13:41.982758 13524 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.13.53.62:40691 I0913 23:13:41.993520 13524 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 15700us I0913 23:13:41.993903 13524 fs_manager.cc:263] Metadata directory not provided I0913 23:13:41.994040 13524 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0913 23:13:41.994410 13524 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0913 23:13:41.994515 13524 server_base.cc:449] Attempting to create new FS layout instead I0913 23:13:41.995426 13524 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568416417781642-13524/minicluster-data/master-0-root/instance: uuid: "afe4b86f03784fdfb3df5e1b8513a3e0" format_stamp: "Formatted at 2019-09-13 23:13:41 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:42.003197 13524 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s I0913 23:13:42.004601 13524 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0913 23:13:42.010991 13524 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0913 23:13:42.011162 13524 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568416417781642-13524/minicluster-data/master-0-root uuid: "afe4b86f03784fdfb3df5e1b8513a3e0" format_stamp: "Formatted at 2019-09-13 23:13:41 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:42.011420 13524 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568416417781642-13524/minicluster-data/master-0-root metadata directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568416417781642-13524/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568416417781642-13524/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0913 23:13:42.081794 13524 rpc_server.cc:205] RPC server started. Bound to: 127.13.53.62:40691 I0913 23:13:42.083839 13524 webserver.cc:374] Webserver started at http://127.13.53.62:39633/ using document root and password file I0913 23:13:42.085690 13832 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0913 23:13:42.092429 13832 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P afe4b86f03784fdfb3df5e1b8513a3e0: Bootstrap starting. I0913 23:13:42.094449 13832 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P afe4b86f03784fdfb3df5e1b8513a3e0: No blocks or log segments found. Creating new log. I0913 23:13:42.097190 13832 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P afe4b86f03784fdfb3df5e1b8513a3e0: No bootstrap required, opened a new log I0913 23:13:42.098467 13832 raft_consensus.cc:340] T 00000000000000000000000000000000 P afe4b86f03784fdfb3df5e1b8513a3e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe4b86f03784fdfb3df5e1b8513a3e0" member_type: VOTER } I0913 23:13:42.098778 13832 raft_consensus.cc:366] T 00000000000000000000000000000000 P afe4b86f03784fdfb3df5e1b8513a3e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0913 23:13:42.098902 13832 raft_consensus.cc:705] T 00000000000000000000000000000000 P afe4b86f03784fdfb3df5e1b8513a3e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afe4b86f03784fdfb3df5e1b8513a3e0, State: Initialized, Role: FOLLOWER I0913 23:13:42.099267 13832 consensus_queue.cc:229] T 00000000000000000000000000000000 P afe4b86f03784fdfb3df5e1b8513a3e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe4b86f03784fdfb3df5e1b8513a3e0" member_type: VOTER } I0913 23:13:42.099650 13832 raft_consensus.cc:380] T 00000000000000000000000000000000 P afe4b86f03784fdfb3df5e1b8513a3e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0913 23:13:42.099789 13832 raft_consensus.cc:472] T 00000000000000000000000000000000 P afe4b86f03784fdfb3df5e1b8513a3e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0913 23:13:42.099957 13832 raft_consensus.cc:2890] T 00000000000000000000000000000000 P afe4b86f03784fdfb3df5e1b8513a3e0 [term 0 FOLLOWER]: Advancing to term 1 I0913 23:13:42.103152 13832 raft_consensus.cc:494] T 00000000000000000000000000000000 P afe4b86f03784fdfb3df5e1b8513a3e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe4b86f03784fdfb3df5e1b8513a3e0" member_type: VOTER } I0913 23:13:42.103547 13832 leader_election.cc:310] T 00000000000000000000000000000000 P afe4b86f03784fdfb3df5e1b8513a3e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afe4b86f03784fdfb3df5e1b8513a3e0; no voters: I0913 23:13:42.104809 13832 leader_election.cc:296] T 00000000000000000000000000000000 P afe4b86f03784fdfb3df5e1b8513a3e0 [CANDIDATE]: Term 1 election: Requested vote from peers I0913 23:13:42.105046 13834 raft_consensus.cc:2644] T 00000000000000000000000000000000 P afe4b86f03784fdfb3df5e1b8513a3e0 [term 1 FOLLOWER]: Leader election won for term 1 I0913 23:13:42.106549 13834 raft_consensus.cc:667] T 00000000000000000000000000000000 P afe4b86f03784fdfb3df5e1b8513a3e0 [term 1 LEADER]: Becoming Leader. State: Replica: afe4b86f03784fdfb3df5e1b8513a3e0, State: Running, Role: LEADER I0913 23:13:42.107020 13834 consensus_queue.cc:206] T 00000000000000000000000000000000 P afe4b86f03784fdfb3df5e1b8513a3e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe4b86f03784fdfb3df5e1b8513a3e0" member_type: VOTER } I0913 23:13:42.109221 13832 sys_catalog.cc:340] T 00000000000000000000000000000000 P afe4b86f03784fdfb3df5e1b8513a3e0 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "afe4b86f03784fdfb3df5e1b8513a3e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe4b86f03784fdfb3df5e1b8513a3e0" member_type: VOTER } } I0913 23:13:42.109400 13835 sys_catalog.cc:340] T 00000000000000000000000000000000 P afe4b86f03784fdfb3df5e1b8513a3e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afe4b86f03784fdfb3df5e1b8513a3e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe4b86f03784fdfb3df5e1b8513a3e0" member_type: VOTER } } I0913 23:13:42.109429 13836 sys_catalog.cc:340] T 00000000000000000000000000000000 P afe4b86f03784fdfb3df5e1b8513a3e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afe4b86f03784fdfb3df5e1b8513a3e0. Latest consensus state: current_term: 1 leader_uuid: "afe4b86f03784fdfb3df5e1b8513a3e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe4b86f03784fdfb3df5e1b8513a3e0" member_type: VOTER } } I0913 23:13:42.109871 13832 sys_catalog.cc:343] T 00000000000000000000000000000000 P afe4b86f03784fdfb3df5e1b8513a3e0 [sys.catalog]: This master's current role is: LEADER I0913 23:13:42.110003 13835 sys_catalog.cc:343] T 00000000000000000000000000000000 P afe4b86f03784fdfb3df5e1b8513a3e0 [sys.catalog]: This master's current role is: LEADER I0913 23:13:42.110170 13836 sys_catalog.cc:343] T 00000000000000000000000000000000 P afe4b86f03784fdfb3df5e1b8513a3e0 [sys.catalog]: This master's current role is: LEADER I0913 23:13:42.111778 13832 sys_catalog.cc:428] T 00000000000000000000000000000000 P afe4b86f03784fdfb3df5e1b8513a3e0 [sys.catalog]: configured and running, proceeding with master startup. I0913 23:13:42.111977 13838 catalog_manager.cc:1071] Loading table and tablet metadata into memory... I0913 23:13:42.114542 13524 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0913 23:13:42.114953 13838 catalog_manager.cc:1082] Initializing Kudu internal certificate authority... I0913 23:13:42.129184 13838 catalog_manager.cc:966] Generated new certificate authority record I0913 23:13:42.130633 13838 catalog_manager.cc:1091] Loading token signing keys... I0913 23:13:42.144937 13838 catalog_manager.cc:4394] T 00000000000000000000000000000000 P afe4b86f03784fdfb3df5e1b8513a3e0: Generated new TSK 0 I0913 23:13:42.158455 13524 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 16200us I0913 23:13:42.158848 13524 fs_manager.cc:263] Metadata directory not provided I0913 23:13:42.158987 13524 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0913 23:13:42.159185 13524 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0913 23:13:42.159288 13524 server_base.cc:449] Attempting to create new FS layout instead I0913 23:13:42.160219 13524 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568416417781642-13524/minicluster-data/ts-0-root/instance: uuid: "63a2a4b091a84abb8413d93b7e2c278d" format_stamp: "Formatted at 2019-09-13 23:13:42 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:42.168035 13524 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.008s sys 0.001s I0913 23:13:42.169307 13524 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0913 23:13:42.175519 13524 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0913 23:13:42.175676 13524 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568416417781642-13524/minicluster-data/ts-0-root uuid: "63a2a4b091a84abb8413d93b7e2c278d" format_stamp: "Formatted at 2019-09-13 23:13:42 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:42.175925 13524 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568416417781642-13524/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568416417781642-13524/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568416417781642-13524/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0913 23:13:42.206051 13524 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0913 23:13:42.206225 13524 ts_tablet_manager.cc:403] Registered 0 tablets I0913 23:13:42.242841 13524 rpc_server.cc:205] RPC server started. Bound to: 127.13.53.1:40411 I0913 23:13:42.244572 13524 webserver.cc:374] Webserver started at http://127.13.53.1:43617/ using document root and password file I0913 23:13:42.260799 13906 heartbeater.cc:322] Connected to a master server at 127.13.53.62:40691 I0913 23:13:42.261106 13906 heartbeater.cc:406] Registering TS with master... I0913 23:13:42.261580 13906 heartbeater.cc:455] Master 127.13.53.62:40691 requested a full tablet report, sending... I0913 23:13:42.263185 13799 ts_manager.cc:161] Registered new tserver with Master: 63a2a4b091a84abb8413d93b7e2c278d (127.13.53.1:40411) I0913 23:13:42.264300 13524 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 16200us I0913 23:13:42.264720 13524 fs_manager.cc:263] Metadata directory not provided I0913 23:13:42.264890 13524 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0913 23:13:42.265110 13524 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0913 23:13:42.265242 13524 server_base.cc:449] Attempting to create new FS layout instead I0913 23:13:42.265413 13799 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39416 I0913 23:13:42.266330 13524 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568416417781642-13524/minicluster-data/ts-1-root/instance: uuid: "e1537387c2ec4f6fb9b300f5bc0c8fc5" format_stamp: "Formatted at 2019-09-13 23:13:42 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:42.275315 13524 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.006s sys 0.003s I0913 23:13:42.276635 13524 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0913 23:13:42.283505 13524 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0913 23:13:42.283680 13524 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568416417781642-13524/minicluster-data/ts-1-root uuid: "e1537387c2ec4f6fb9b300f5bc0c8fc5" format_stamp: "Formatted at 2019-09-13 23:13:42 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:42.283892 13524 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568416417781642-13524/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568416417781642-13524/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568416417781642-13524/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0913 23:13:42.342620 13524 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0913 23:13:42.342780 13524 ts_tablet_manager.cc:403] Registered 0 tablets I0913 23:13:42.376070 13524 rpc_server.cc:205] RPC server started. Bound to: 127.13.53.2:37221 I0913 23:13:42.377921 13524 webserver.cc:374] Webserver started at http://127.13.53.2:35467/ using document root and password file I0913 23:13:42.391106 13974 heartbeater.cc:322] Connected to a master server at 127.13.53.62:40691 I0913 23:13:42.391386 13974 heartbeater.cc:406] Registering TS with master... I0913 23:13:42.391796 13974 heartbeater.cc:455] Master 127.13.53.62:40691 requested a full tablet report, sending... I0913 23:13:42.393234 13799 ts_manager.cc:161] Registered new tserver with Master: e1537387c2ec4f6fb9b300f5bc0c8fc5 (127.13.53.2:37221) I0913 23:13:42.394563 13799 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39418 I0913 23:13:42.396582 13524 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 16200us I0913 23:13:42.396929 13524 fs_manager.cc:263] Metadata directory not provided I0913 23:13:42.397066 13524 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0913 23:13:42.397243 13524 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0913 23:13:42.397347 13524 server_base.cc:449] Attempting to create new FS layout instead I0913 23:13:42.398285 13524 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568416417781642-13524/minicluster-data/ts-2-root/instance: uuid: "220ce5b215394d149756f248d7b323de" format_stamp: "Formatted at 2019-09-13 23:13:42 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:42.406880 13524 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s I0913 23:13:42.408291 13524 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0913 23:13:42.415411 13524 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0913 23:13:42.415566 13524 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568416417781642-13524/minicluster-data/ts-2-root uuid: "220ce5b215394d149756f248d7b323de" format_stamp: "Formatted at 2019-09-13 23:13:42 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:42.415788 13524 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568416417781642-13524/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568416417781642-13524/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568416417781642-13524/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0913 23:13:42.449087 13524 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0913 23:13:42.449255 13524 ts_tablet_manager.cc:403] Registered 0 tablets I0913 23:13:42.515556 13524 rpc_server.cc:205] RPC server started. Bound to: 127.13.53.3:34097 I0913 23:13:42.517833 13524 webserver.cc:374] Webserver started at http://127.13.53.3:33513/ using document root and password file I0913 23:13:42.529976 14041 heartbeater.cc:322] Connected to a master server at 127.13.53.62:40691 I0913 23:13:42.530299 14041 heartbeater.cc:406] Registering TS with master... I0913 23:13:42.530741 14041 heartbeater.cc:455] Master 127.13.53.62:40691 requested a full tablet report, sending... I0913 23:13:42.531994 13799 ts_manager.cc:161] Registered new tserver with Master: 220ce5b215394d149756f248d7b323de (127.13.53.3:34097) I0913 23:13:42.532785 13524 internal_mini_cluster.cc:328] 3 TS(s) registered with all masters after 0.012033312s I0913 23:13:42.533407 13799 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39420 I0913 23:13:42.555176 13799 catalog_manager.cc:1425] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39422: name: "test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { } } W0913 23:13:42.555971 13799 catalog_manager.cc:1590] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0913 23:13:42.633185 14008 tablet_service.cc:1022] Processing CreateTablet for tablet e46d166a511d45f3b10efe8aabd64cf7 (table=test-table [id=5c37c10047004656acdca7478b8b45e5]), partition= I0913 23:13:42.633868 14008 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet e46d166a511d45f3b10efe8aabd64cf7. 1 dirs total, 0 dirs full, 0 dirs failed I0913 23:13:42.634124 13941 tablet_service.cc:1022] Processing CreateTablet for tablet e46d166a511d45f3b10efe8aabd64cf7 (table=test-table [id=5c37c10047004656acdca7478b8b45e5]), partition= I0913 23:13:42.634726 13941 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet e46d166a511d45f3b10efe8aabd64cf7. 1 dirs total, 0 dirs full, 0 dirs failed I0913 23:13:42.643544 14057 tablet_bootstrap.cc:439] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5: Bootstrap starting. I0913 23:13:42.642947 14056 tablet_bootstrap.cc:439] T e46d166a511d45f3b10efe8aabd64cf7 P 220ce5b215394d149756f248d7b323de: Bootstrap starting. I0913 23:13:42.645931 13873 tablet_service.cc:1022] Processing CreateTablet for tablet e46d166a511d45f3b10efe8aabd64cf7 (table=test-table [id=5c37c10047004656acdca7478b8b45e5]), partition= I0913 23:13:42.646706 13873 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet e46d166a511d45f3b10efe8aabd64cf7. 1 dirs total, 0 dirs full, 0 dirs failed I0913 23:13:42.647228 14057 tablet_bootstrap.cc:590] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5: No blocks or log segments found. Creating new log. I0913 23:13:42.647725 14056 tablet_bootstrap.cc:590] T e46d166a511d45f3b10efe8aabd64cf7 P 220ce5b215394d149756f248d7b323de: No blocks or log segments found. Creating new log. I0913 23:13:42.663362 14057 tablet_bootstrap.cc:439] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5: No bootstrap required, opened a new log I0913 23:13:42.663841 14060 tablet_bootstrap.cc:439] T e46d166a511d45f3b10efe8aabd64cf7 P 63a2a4b091a84abb8413d93b7e2c278d: Bootstrap starting. I0913 23:13:42.663938 14057 ts_tablet_manager.cc:1085] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.012s I0913 23:13:42.663362 14056 tablet_bootstrap.cc:439] T e46d166a511d45f3b10efe8aabd64cf7 P 220ce5b215394d149756f248d7b323de: No bootstrap required, opened a new log I0913 23:13:42.666015 14056 ts_tablet_manager.cc:1085] T e46d166a511d45f3b10efe8aabd64cf7 P 220ce5b215394d149756f248d7b323de: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.014s I0913 23:13:42.666491 14057 raft_consensus.cc:340] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220ce5b215394d149756f248d7b323de" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 34097 } } peers { permanent_uuid: "e1537387c2ec4f6fb9b300f5bc0c8fc5" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 37221 } } peers { permanent_uuid: "63a2a4b091a84abb8413d93b7e2c278d" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 40411 } } I0913 23:13:42.667110 14057 raft_consensus.cc:366] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0913 23:13:42.667284 14057 raft_consensus.cc:705] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1537387c2ec4f6fb9b300f5bc0c8fc5, State: Initialized, Role: FOLLOWER I0913 23:13:42.667737 14056 raft_consensus.cc:340] T e46d166a511d45f3b10efe8aabd64cf7 P 220ce5b215394d149756f248d7b323de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220ce5b215394d149756f248d7b323de" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 34097 } } peers { permanent_uuid: "e1537387c2ec4f6fb9b300f5bc0c8fc5" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 37221 } } peers { permanent_uuid: "63a2a4b091a84abb8413d93b7e2c278d" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 40411 } } I0913 23:13:42.667726 14057 consensus_queue.cc:229] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220ce5b215394d149756f248d7b323de" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 34097 } } peers { permanent_uuid: "e1537387c2ec4f6fb9b300f5bc0c8fc5" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 37221 } } peers { permanent_uuid: "63a2a4b091a84abb8413d93b7e2c278d" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 40411 } } I0913 23:13:42.668309 14056 raft_consensus.cc:366] T e46d166a511d45f3b10efe8aabd64cf7 P 220ce5b215394d149756f248d7b323de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0913 23:13:42.668576 14056 raft_consensus.cc:705] T e46d166a511d45f3b10efe8aabd64cf7 P 220ce5b215394d149756f248d7b323de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 220ce5b215394d149756f248d7b323de, State: Initialized, Role: FOLLOWER I0913 23:13:42.669136 14056 consensus_queue.cc:229] T e46d166a511d45f3b10efe8aabd64cf7 P 220ce5b215394d149756f248d7b323de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220ce5b215394d149756f248d7b323de" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 34097 } } peers { permanent_uuid: "e1537387c2ec4f6fb9b300f5bc0c8fc5" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 37221 } } peers { permanent_uuid: "63a2a4b091a84abb8413d93b7e2c278d" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 40411 } } I0913 23:13:42.670831 14060 tablet_bootstrap.cc:590] T e46d166a511d45f3b10efe8aabd64cf7 P 63a2a4b091a84abb8413d93b7e2c278d: No blocks or log segments found. Creating new log. I0913 23:13:42.670902 13974 heartbeater.cc:447] Master 127.13.53.62:40691 was elected leader, sending a full tablet report... I0913 23:13:42.672593 14057 ts_tablet_manager.cc:1113] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s I0913 23:13:42.673285 14041 heartbeater.cc:447] Master 127.13.53.62:40691 was elected leader, sending a full tablet report... I0913 23:13:42.676520 14056 ts_tablet_manager.cc:1113] T e46d166a511d45f3b10efe8aabd64cf7 P 220ce5b215394d149756f248d7b323de: Time spent starting tablet: real 0.010s user 0.003s sys 0.003s I0913 23:13:42.678838 14060 tablet_bootstrap.cc:439] T e46d166a511d45f3b10efe8aabd64cf7 P 63a2a4b091a84abb8413d93b7e2c278d: No bootstrap required, opened a new log I0913 23:13:42.679200 14060 ts_tablet_manager.cc:1085] T e46d166a511d45f3b10efe8aabd64cf7 P 63a2a4b091a84abb8413d93b7e2c278d: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s I0913 23:13:42.680541 14060 raft_consensus.cc:340] T e46d166a511d45f3b10efe8aabd64cf7 P 63a2a4b091a84abb8413d93b7e2c278d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220ce5b215394d149756f248d7b323de" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 34097 } } peers { permanent_uuid: "e1537387c2ec4f6fb9b300f5bc0c8fc5" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 37221 } } peers { permanent_uuid: "63a2a4b091a84abb8413d93b7e2c278d" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 40411 } } I0913 23:13:42.680994 14060 raft_consensus.cc:366] T e46d166a511d45f3b10efe8aabd64cf7 P 63a2a4b091a84abb8413d93b7e2c278d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0913 23:13:42.681161 14060 raft_consensus.cc:705] T e46d166a511d45f3b10efe8aabd64cf7 P 63a2a4b091a84abb8413d93b7e2c278d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63a2a4b091a84abb8413d93b7e2c278d, State: Initialized, Role: FOLLOWER I0913 23:13:42.681557 14060 consensus_queue.cc:229] T e46d166a511d45f3b10efe8aabd64cf7 P 63a2a4b091a84abb8413d93b7e2c278d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220ce5b215394d149756f248d7b323de" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 34097 } } peers { permanent_uuid: "e1537387c2ec4f6fb9b300f5bc0c8fc5" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 37221 } } peers { permanent_uuid: "63a2a4b091a84abb8413d93b7e2c278d" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 40411 } } I0913 23:13:42.683177 13906 heartbeater.cc:447] Master 127.13.53.62:40691 was elected leader, sending a full tablet report... I0913 23:13:42.683405 14060 ts_tablet_manager.cc:1113] T e46d166a511d45f3b10efe8aabd64cf7 P 63a2a4b091a84abb8413d93b7e2c278d: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s I0913 23:13:42.883939 14064 raft_consensus.cc:472] T e46d166a511d45f3b10efe8aabd64cf7 P 63a2a4b091a84abb8413d93b7e2c278d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0913 23:13:42.884395 14064 raft_consensus.cc:494] T e46d166a511d45f3b10efe8aabd64cf7 P 63a2a4b091a84abb8413d93b7e2c278d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220ce5b215394d149756f248d7b323de" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 34097 } } peers { permanent_uuid: "e1537387c2ec4f6fb9b300f5bc0c8fc5" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 37221 } } peers { permanent_uuid: "63a2a4b091a84abb8413d93b7e2c278d" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 40411 } } I0913 23:13:42.885949 14064 leader_election.cc:296] T e46d166a511d45f3b10efe8aabd64cf7 P 63a2a4b091a84abb8413d93b7e2c278d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 220ce5b215394d149756f248d7b323de (127.13.53.3:34097), e1537387c2ec4f6fb9b300f5bc0c8fc5 (127.13.53.2:37221) I0913 23:13:42.898608 14018 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e46d166a511d45f3b10efe8aabd64cf7" candidate_uuid: "63a2a4b091a84abb8413d93b7e2c278d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "220ce5b215394d149756f248d7b323de" is_pre_election: true I0913 23:13:42.899400 14018 raft_consensus.cc:2350] T e46d166a511d45f3b10efe8aabd64cf7 P 220ce5b215394d149756f248d7b323de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63a2a4b091a84abb8413d93b7e2c278d in term 0. I0913 23:13:42.899708 13951 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e46d166a511d45f3b10efe8aabd64cf7" candidate_uuid: "63a2a4b091a84abb8413d93b7e2c278d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1537387c2ec4f6fb9b300f5bc0c8fc5" is_pre_election: true I0913 23:13:42.900205 13951 raft_consensus.cc:2350] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63a2a4b091a84abb8413d93b7e2c278d in term 0. I0913 23:13:42.900368 13845 leader_election.cc:310] T e46d166a511d45f3b10efe8aabd64cf7 P 63a2a4b091a84abb8413d93b7e2c278d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 220ce5b215394d149756f248d7b323de, 63a2a4b091a84abb8413d93b7e2c278d; no voters: I0913 23:13:42.901124 14064 raft_consensus.cc:2644] T e46d166a511d45f3b10efe8aabd64cf7 P 63a2a4b091a84abb8413d93b7e2c278d [term 0 FOLLOWER]: Leader pre-election won for term 1 I0913 23:13:42.901324 14064 raft_consensus.cc:472] T e46d166a511d45f3b10efe8aabd64cf7 P 63a2a4b091a84abb8413d93b7e2c278d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0913 23:13:42.901530 14064 raft_consensus.cc:2890] T e46d166a511d45f3b10efe8aabd64cf7 P 63a2a4b091a84abb8413d93b7e2c278d [term 0 FOLLOWER]: Advancing to term 1 I0913 23:13:42.905185 14064 raft_consensus.cc:494] T e46d166a511d45f3b10efe8aabd64cf7 P 63a2a4b091a84abb8413d93b7e2c278d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220ce5b215394d149756f248d7b323de" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 34097 } } peers { permanent_uuid: "e1537387c2ec4f6fb9b300f5bc0c8fc5" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 37221 } } peers { permanent_uuid: "63a2a4b091a84abb8413d93b7e2c278d" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 40411 } } I0913 23:13:42.906327 14064 leader_election.cc:296] T e46d166a511d45f3b10efe8aabd64cf7 P 63a2a4b091a84abb8413d93b7e2c278d [CANDIDATE]: Term 1 election: Requested vote from peers 220ce5b215394d149756f248d7b323de (127.13.53.3:34097), e1537387c2ec4f6fb9b300f5bc0c8fc5 (127.13.53.2:37221) I0913 23:13:42.907024 14018 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e46d166a511d45f3b10efe8aabd64cf7" candidate_uuid: "63a2a4b091a84abb8413d93b7e2c278d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "220ce5b215394d149756f248d7b323de" I0913 23:13:42.907093 13951 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e46d166a511d45f3b10efe8aabd64cf7" candidate_uuid: "63a2a4b091a84abb8413d93b7e2c278d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1537387c2ec4f6fb9b300f5bc0c8fc5" I0913 23:13:42.907516 14018 raft_consensus.cc:2890] T e46d166a511d45f3b10efe8aabd64cf7 P 220ce5b215394d149756f248d7b323de [term 0 FOLLOWER]: Advancing to term 1 I0913 23:13:42.907586 13951 raft_consensus.cc:2890] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5 [term 0 FOLLOWER]: Advancing to term 1 I0913 23:13:42.911411 13951 raft_consensus.cc:2350] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63a2a4b091a84abb8413d93b7e2c278d in term 1. I0913 23:13:42.911453 14018 raft_consensus.cc:2350] T e46d166a511d45f3b10efe8aabd64cf7 P 220ce5b215394d149756f248d7b323de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63a2a4b091a84abb8413d93b7e2c278d in term 1. I0913 23:13:42.912313 13844 leader_election.cc:310] T e46d166a511d45f3b10efe8aabd64cf7 P 63a2a4b091a84abb8413d93b7e2c278d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63a2a4b091a84abb8413d93b7e2c278d, e1537387c2ec4f6fb9b300f5bc0c8fc5; no voters: I0913 23:13:42.912912 14064 raft_consensus.cc:2644] T e46d166a511d45f3b10efe8aabd64cf7 P 63a2a4b091a84abb8413d93b7e2c278d [term 1 FOLLOWER]: Leader election won for term 1 I0913 23:13:42.913661 14064 raft_consensus.cc:667] T e46d166a511d45f3b10efe8aabd64cf7 P 63a2a4b091a84abb8413d93b7e2c278d [term 1 LEADER]: Becoming Leader. State: Replica: 63a2a4b091a84abb8413d93b7e2c278d, State: Running, Role: LEADER I0913 23:13:42.914172 14064 consensus_queue.cc:206] T e46d166a511d45f3b10efe8aabd64cf7 P 63a2a4b091a84abb8413d93b7e2c278d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220ce5b215394d149756f248d7b323de" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 34097 } } peers { permanent_uuid: "e1537387c2ec4f6fb9b300f5bc0c8fc5" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 37221 } } peers { permanent_uuid: "63a2a4b091a84abb8413d93b7e2c278d" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 40411 } } I0913 23:13:42.919100 13798 catalog_manager.cc:4079] T e46d166a511d45f3b10efe8aabd64cf7 P 63a2a4b091a84abb8413d93b7e2c278d reported cstate change: term changed from 0 to 1, leader changed from to 63a2a4b091a84abb8413d93b7e2c278d (127.13.53.1). New cstate: current_term: 1 leader_uuid: "63a2a4b091a84abb8413d93b7e2c278d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220ce5b215394d149756f248d7b323de" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 34097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1537387c2ec4f6fb9b300f5bc0c8fc5" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 37221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63a2a4b091a84abb8413d93b7e2c278d" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 40411 } health_report { overall_health: HEALTHY } } } I0913 23:13:43.353910 14064 consensus_queue.cc:984] T e46d166a511d45f3b10efe8aabd64cf7 P 63a2a4b091a84abb8413d93b7e2c278d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1537387c2ec4f6fb9b300f5bc0c8fc5" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 37221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0913 23:13:43.363046 14073 consensus_queue.cc:984] T e46d166a511d45f3b10efe8aabd64cf7 P 63a2a4b091a84abb8413d93b7e2c278d [LEADER]: Connected to new peer: Peer: permanent_uuid: "220ce5b215394d149756f248d7b323de" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 34097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0913 23:13:43.975024 13524 tablet_replica.cc:273] T e46d166a511d45f3b10efe8aabd64cf7 P 63a2a4b091a84abb8413d93b7e2c278d: stopping tablet replica I0913 23:13:43.975404 13524 raft_consensus.cc:2147] T e46d166a511d45f3b10efe8aabd64cf7 P 63a2a4b091a84abb8413d93b7e2c278d [term 1 LEADER]: Raft consensus shutting down. I0913 23:13:43.975941 13524 raft_consensus.cc:2174] T e46d166a511d45f3b10efe8aabd64cf7 P 63a2a4b091a84abb8413d93b7e2c278d [term 1 FOLLOWER]: Raft consensus is shut down! I0913 23:13:45.251713 14078 raft_consensus.cc:472] T e46d166a511d45f3b10efe8aabd64cf7 P 220ce5b215394d149756f248d7b323de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 63a2a4b091a84abb8413d93b7e2c278d) I0913 23:13:45.252101 14078 raft_consensus.cc:494] T e46d166a511d45f3b10efe8aabd64cf7 P 220ce5b215394d149756f248d7b323de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220ce5b215394d149756f248d7b323de" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 34097 } } peers { permanent_uuid: "e1537387c2ec4f6fb9b300f5bc0c8fc5" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 37221 } } peers { permanent_uuid: "63a2a4b091a84abb8413d93b7e2c278d" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 40411 } } I0913 23:13:45.253490 14078 leader_election.cc:296] T e46d166a511d45f3b10efe8aabd64cf7 P 220ce5b215394d149756f248d7b323de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1537387c2ec4f6fb9b300f5bc0c8fc5 (127.13.53.2:37221), 63a2a4b091a84abb8413d93b7e2c278d (127.13.53.1:40411) I0913 23:13:45.269640 13883 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e46d166a511d45f3b10efe8aabd64cf7" candidate_uuid: "220ce5b215394d149756f248d7b323de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "63a2a4b091a84abb8413d93b7e2c278d" is_pre_election: true I0913 23:13:45.269896 13951 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e46d166a511d45f3b10efe8aabd64cf7" candidate_uuid: "220ce5b215394d149756f248d7b323de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e1537387c2ec4f6fb9b300f5bc0c8fc5" is_pre_election: true I0913 23:13:45.270570 13951 raft_consensus.cc:2311] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 220ce5b215394d149756f248d7b323de for term 2 because replica is either leader or believes a valid leader to be alive. W0913 23:13:45.271212 13982 leader_election.cc:348] T e46d166a511d45f3b10efe8aabd64cf7 P 220ce5b215394d149756f248d7b323de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63a2a4b091a84abb8413d93b7e2c278d (127.13.53.1:40411): Illegal state: cannot vote while shut down I0913 23:13:45.271647 13979 leader_election.cc:310] T e46d166a511d45f3b10efe8aabd64cf7 P 220ce5b215394d149756f248d7b323de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 220ce5b215394d149756f248d7b323de; no voters: 63a2a4b091a84abb8413d93b7e2c278d, e1537387c2ec4f6fb9b300f5bc0c8fc5 I0913 23:13:45.272212 14078 raft_consensus.cc:2592] T e46d166a511d45f3b10efe8aabd64cf7 P 220ce5b215394d149756f248d7b323de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0913 23:13:45.405818 14083 raft_consensus.cc:472] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 63a2a4b091a84abb8413d93b7e2c278d) I0913 23:13:45.406273 14083 raft_consensus.cc:494] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220ce5b215394d149756f248d7b323de" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 34097 } } peers { permanent_uuid: "e1537387c2ec4f6fb9b300f5bc0c8fc5" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 37221 } } peers { permanent_uuid: "63a2a4b091a84abb8413d93b7e2c278d" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 40411 } } I0913 23:13:45.407582 14083 leader_election.cc:296] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 220ce5b215394d149756f248d7b323de (127.13.53.3:34097), 63a2a4b091a84abb8413d93b7e2c278d (127.13.53.1:40411) I0913 23:13:45.422756 14018 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e46d166a511d45f3b10efe8aabd64cf7" candidate_uuid: "e1537387c2ec4f6fb9b300f5bc0c8fc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "220ce5b215394d149756f248d7b323de" is_pre_election: true I0913 23:13:45.423352 14018 raft_consensus.cc:2350] T e46d166a511d45f3b10efe8aabd64cf7 P 220ce5b215394d149756f248d7b323de [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1537387c2ec4f6fb9b300f5bc0c8fc5 in term 1. I0913 23:13:45.424302 13883 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e46d166a511d45f3b10efe8aabd64cf7" candidate_uuid: "e1537387c2ec4f6fb9b300f5bc0c8fc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "63a2a4b091a84abb8413d93b7e2c278d" is_pre_election: true I0913 23:13:45.424507 13913 leader_election.cc:310] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 220ce5b215394d149756f248d7b323de, e1537387c2ec4f6fb9b300f5bc0c8fc5; no voters: I0913 23:13:45.425230 14083 raft_consensus.cc:2644] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0913 23:13:45.425510 14083 raft_consensus.cc:472] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 63a2a4b091a84abb8413d93b7e2c278d) W0913 23:13:45.425599 13915 leader_election.cc:348] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63a2a4b091a84abb8413d93b7e2c278d (127.13.53.1:40411): Illegal state: cannot vote while shut down I0913 23:13:45.425796 14083 raft_consensus.cc:2890] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5 [term 1 FOLLOWER]: Advancing to term 2 I0913 23:13:45.429819 14083 raft_consensus.cc:494] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220ce5b215394d149756f248d7b323de" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 34097 } } peers { permanent_uuid: "e1537387c2ec4f6fb9b300f5bc0c8fc5" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 37221 } } peers { permanent_uuid: "63a2a4b091a84abb8413d93b7e2c278d" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 40411 } } I0913 23:13:45.430958 14083 leader_election.cc:296] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5 [CANDIDATE]: Term 2 election: Requested vote from peers 220ce5b215394d149756f248d7b323de (127.13.53.3:34097), 63a2a4b091a84abb8413d93b7e2c278d (127.13.53.1:40411) I0913 23:13:45.431866 13883 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e46d166a511d45f3b10efe8aabd64cf7" candidate_uuid: "e1537387c2ec4f6fb9b300f5bc0c8fc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "63a2a4b091a84abb8413d93b7e2c278d" I0913 23:13:45.432170 14018 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "e46d166a511d45f3b10efe8aabd64cf7" candidate_uuid: "e1537387c2ec4f6fb9b300f5bc0c8fc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "220ce5b215394d149756f248d7b323de" I0913 23:13:45.432600 14018 raft_consensus.cc:2890] T e46d166a511d45f3b10efe8aabd64cf7 P 220ce5b215394d149756f248d7b323de [term 1 FOLLOWER]: Advancing to term 2 W0913 23:13:45.433028 13915 leader_election.cc:348] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 63a2a4b091a84abb8413d93b7e2c278d (127.13.53.1:40411): Illegal state: cannot vote while shut down I0913 23:13:45.436053 14018 raft_consensus.cc:2350] T e46d166a511d45f3b10efe8aabd64cf7 P 220ce5b215394d149756f248d7b323de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1537387c2ec4f6fb9b300f5bc0c8fc5 in term 2. I0913 23:13:45.436818 13913 leader_election.cc:310] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 220ce5b215394d149756f248d7b323de, e1537387c2ec4f6fb9b300f5bc0c8fc5; no voters: 63a2a4b091a84abb8413d93b7e2c278d I0913 23:13:45.437409 14083 raft_consensus.cc:2644] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5 [term 2 FOLLOWER]: Leader election won for term 2 I0913 23:13:45.438623 14083 raft_consensus.cc:667] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5 [term 2 LEADER]: Becoming Leader. State: Replica: e1537387c2ec4f6fb9b300f5bc0c8fc5, State: Running, Role: LEADER I0913 23:13:45.439038 14083 consensus_queue.cc:206] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220ce5b215394d149756f248d7b323de" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 34097 } } peers { permanent_uuid: "e1537387c2ec4f6fb9b300f5bc0c8fc5" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 37221 } } peers { permanent_uuid: "63a2a4b091a84abb8413d93b7e2c278d" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 40411 } } I0913 23:13:45.827994 13915 consensus_queue.cc:544] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5 [LEADER]: The tablet replica hosted on peer 63a2a4b091a84abb8413d93b7e2c278d has failed W0913 23:13:45.828307 13915 consensus_peers.cc:458] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5 -> Peer 63a2a4b091a84abb8413d93b7e2c278d (127.13.53.1:40411): Couldn't send request to peer 63a2a4b091a84abb8413d93b7e2c278d. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry. I0913 23:13:45.916527 14018 raft_consensus.cc:1184] T e46d166a511d45f3b10efe8aabd64cf7 P 220ce5b215394d149756f248d7b323de [term 2 FOLLOWER]: Refusing update from remote peer e1537387c2ec4f6fb9b300f5bc0c8fc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch) I0913 23:13:45.917768 14087 consensus_queue.cc:984] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "220ce5b215394d149756f248d7b323de" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 34097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0913 23:13:46.011720 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:13:46.035449 13524 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0913 23:13:46.051424 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:13:46.051614 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:13:46.051910 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:13:46.057565 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:13:46.071753 13524 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0913 23:13:46.072041 13524 tablet_replica.cc:273] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5: stopping tablet replica I0913 23:13:46.072332 13524 raft_consensus.cc:2147] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5 [term 2 LEADER]: Raft consensus shutting down. I0913 23:13:46.072880 13524 raft_consensus.cc:2174] T e46d166a511d45f3b10efe8aabd64cf7 P e1537387c2ec4f6fb9b300f5bc0c8fc5 [term 2 FOLLOWER]: Raft consensus is shut down! I0913 23:13:46.081923 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:13:46.082165 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:13:46.082453 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:13:46.087770 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:13:46.100759 13524 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0913 23:13:46.101032 13524 tablet_replica.cc:273] T e46d166a511d45f3b10efe8aabd64cf7 P 220ce5b215394d149756f248d7b323de: stopping tablet replica I0913 23:13:46.101320 13524 raft_consensus.cc:2147] T e46d166a511d45f3b10efe8aabd64cf7 P 220ce5b215394d149756f248d7b323de [term 2 FOLLOWER]: Raft consensus shutting down. I0913 23:13:46.101562 13524 raft_consensus.cc:2174] T e46d166a511d45f3b10efe8aabd64cf7 P 220ce5b215394d149756f248d7b323de [term 2 FOLLOWER]: Raft consensus is shut down! I0913 23:13:46.110605 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:13:46.110759 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:13:46.111006 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:13:46.116518 13524 master.cc:247] Master@127.13.53.62:40691 shutting down... I0913 23:13:46.128119 13524 raft_consensus.cc:2147] T 00000000000000000000000000000000 P afe4b86f03784fdfb3df5e1b8513a3e0 [term 1 LEADER]: Raft consensus shutting down. I0913 23:13:46.128479 13524 raft_consensus.cc:2174] T 00000000000000000000000000000000 P afe4b86f03784fdfb3df5e1b8513a3e0 [term 1 FOLLOWER]: Raft consensus is shut down! I0913 23:13:46.128669 13524 tablet_replica.cc:273] T 00000000000000000000000000000000 P afe4b86f03784fdfb3df5e1b8513a3e0: stopping tablet replica I0913 23:13:46.144838 13524 master.cc:258] Master@127.13.53.62:40691 shutdown complete. [ OK ] TsTabletManagerITest.ReportOnReplicaHealthStatus (4180 ms) [ RUN ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters I0913 23:13:46.162578 13524 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.13.53.62:36199 I0913 23:13:46.173262 13524 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 18200us I0913 23:13:46.173650 13524 fs_manager.cc:263] Metadata directory not provided I0913 23:13:46.173787 13524 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0913 23:13:46.174006 13524 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0913 23:13:46.174137 13524 server_base.cc:449] Attempting to create new FS layout instead I0913 23:13:46.175061 13524 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568416417781642-13524/minicluster-data/master-0-root/instance: uuid: "d3e419641886423098d5a67c269bee5f" format_stamp: "Formatted at 2019-09-13 23:13:46 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:46.183053 13524 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.005s sys 0.003s I0913 23:13:46.184398 13524 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0913 23:13:46.190636 13524 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0913 23:13:46.190788 13524 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568416417781642-13524/minicluster-data/master-0-root uuid: "d3e419641886423098d5a67c269bee5f" format_stamp: "Formatted at 2019-09-13 23:13:46 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:46.191079 13524 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568416417781642-13524/minicluster-data/master-0-root metadata directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568416417781642-13524/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568416417781642-13524/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0913 23:13:46.267300 13524 rpc_server.cc:205] RPC server started. Bound to: 127.13.53.62:36199 I0913 23:13:46.268937 13524 webserver.cc:374] Webserver started at http://127.13.53.62:34421/ using document root and password file I0913 23:13:46.270591 14151 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0913 23:13:46.277935 14151 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P d3e419641886423098d5a67c269bee5f: Bootstrap starting. I0913 23:13:46.280076 14151 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P d3e419641886423098d5a67c269bee5f: No blocks or log segments found. Creating new log. I0913 23:13:46.282856 14151 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P d3e419641886423098d5a67c269bee5f: No bootstrap required, opened a new log I0913 23:13:46.284140 14151 raft_consensus.cc:340] T 00000000000000000000000000000000 P d3e419641886423098d5a67c269bee5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3e419641886423098d5a67c269bee5f" member_type: VOTER } I0913 23:13:46.284467 14151 raft_consensus.cc:366] T 00000000000000000000000000000000 P d3e419641886423098d5a67c269bee5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0913 23:13:46.284612 14151 raft_consensus.cc:705] T 00000000000000000000000000000000 P d3e419641886423098d5a67c269bee5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3e419641886423098d5a67c269bee5f, State: Initialized, Role: FOLLOWER I0913 23:13:46.284987 14151 consensus_queue.cc:229] T 00000000000000000000000000000000 P d3e419641886423098d5a67c269bee5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3e419641886423098d5a67c269bee5f" member_type: VOTER } I0913 23:13:46.285353 14151 raft_consensus.cc:380] T 00000000000000000000000000000000 P d3e419641886423098d5a67c269bee5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0913 23:13:46.285502 14151 raft_consensus.cc:472] T 00000000000000000000000000000000 P d3e419641886423098d5a67c269bee5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0913 23:13:46.285681 14151 raft_consensus.cc:2890] T 00000000000000000000000000000000 P d3e419641886423098d5a67c269bee5f [term 0 FOLLOWER]: Advancing to term 1 I0913 23:13:46.288920 14151 raft_consensus.cc:494] T 00000000000000000000000000000000 P d3e419641886423098d5a67c269bee5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3e419641886423098d5a67c269bee5f" member_type: VOTER } I0913 23:13:46.289309 14151 leader_election.cc:310] T 00000000000000000000000000000000 P d3e419641886423098d5a67c269bee5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3e419641886423098d5a67c269bee5f; no voters: I0913 23:13:46.290611 14151 leader_election.cc:296] T 00000000000000000000000000000000 P d3e419641886423098d5a67c269bee5f [CANDIDATE]: Term 1 election: Requested vote from peers I0913 23:13:46.290819 14153 raft_consensus.cc:2644] T 00000000000000000000000000000000 P d3e419641886423098d5a67c269bee5f [term 1 FOLLOWER]: Leader election won for term 1 I0913 23:13:46.292354 14153 raft_consensus.cc:667] T 00000000000000000000000000000000 P d3e419641886423098d5a67c269bee5f [term 1 LEADER]: Becoming Leader. State: Replica: d3e419641886423098d5a67c269bee5f, State: Running, Role: LEADER I0913 23:13:46.292912 14153 consensus_queue.cc:206] T 00000000000000000000000000000000 P d3e419641886423098d5a67c269bee5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3e419641886423098d5a67c269bee5f" member_type: VOTER } I0913 23:13:46.295181 14151 sys_catalog.cc:340] T 00000000000000000000000000000000 P d3e419641886423098d5a67c269bee5f [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "d3e419641886423098d5a67c269bee5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3e419641886423098d5a67c269bee5f" member_type: VOTER } } I0913 23:13:46.295377 14154 sys_catalog.cc:340] T 00000000000000000000000000000000 P d3e419641886423098d5a67c269bee5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3e419641886423098d5a67c269bee5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3e419641886423098d5a67c269bee5f" member_type: VOTER } } I0913 23:13:46.295763 14151 sys_catalog.cc:343] T 00000000000000000000000000000000 P d3e419641886423098d5a67c269bee5f [sys.catalog]: This master's current role is: LEADER I0913 23:13:46.296039 14154 sys_catalog.cc:343] T 00000000000000000000000000000000 P d3e419641886423098d5a67c269bee5f [sys.catalog]: This master's current role is: LEADER I0913 23:13:46.295365 14155 sys_catalog.cc:340] T 00000000000000000000000000000000 P d3e419641886423098d5a67c269bee5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3e419641886423098d5a67c269bee5f. Latest consensus state: current_term: 1 leader_uuid: "d3e419641886423098d5a67c269bee5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3e419641886423098d5a67c269bee5f" member_type: VOTER } } I0913 23:13:46.296350 14155 sys_catalog.cc:343] T 00000000000000000000000000000000 P d3e419641886423098d5a67c269bee5f [sys.catalog]: This master's current role is: LEADER I0913 23:13:46.297858 14151 sys_catalog.cc:428] T 00000000000000000000000000000000 P d3e419641886423098d5a67c269bee5f [sys.catalog]: configured and running, proceeding with master startup. I0913 23:13:46.297994 14158 catalog_manager.cc:1071] Loading table and tablet metadata into memory... I0913 23:13:46.299090 13524 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0913 23:13:46.300597 14158 catalog_manager.cc:1082] Initializing Kudu internal certificate authority... I0913 23:13:46.355944 14158 catalog_manager.cc:966] Generated new certificate authority record I0913 23:13:46.357447 14158 catalog_manager.cc:1091] Loading token signing keys... I0913 23:13:46.372067 14158 catalog_manager.cc:4394] T 00000000000000000000000000000000 P d3e419641886423098d5a67c269bee5f: Generated new TSK 0 I0913 23:13:46.439810 13524 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 18200us I0913 23:13:46.440183 13524 fs_manager.cc:263] Metadata directory not provided I0913 23:13:46.440325 13524 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0913 23:13:46.440513 13524 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0913 23:13:46.440618 13524 server_base.cc:449] Attempting to create new FS layout instead I0913 23:13:46.441540 13524 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568416417781642-13524/minicluster-data/ts-0-root/instance: uuid: "aec0bff686fa48748eec67f1005139ac" format_stamp: "Formatted at 2019-09-13 23:13:46 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:46.449303 13524 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.005s sys 0.005s I0913 23:13:46.450768 13524 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0913 23:13:46.457324 13524 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0913 23:13:46.457509 13524 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568416417781642-13524/minicluster-data/ts-0-root uuid: "aec0bff686fa48748eec67f1005139ac" format_stamp: "Formatted at 2019-09-13 23:13:46 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:46.457757 13524 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568416417781642-13524/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568416417781642-13524/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568416417781642-13524/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0913 23:13:46.476275 13524 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0913 23:13:46.476444 13524 ts_tablet_manager.cc:403] Registered 0 tablets I0913 23:13:46.511394 13524 rpc_server.cc:205] RPC server started. Bound to: 127.13.53.1:34579 I0913 23:13:46.513021 13524 webserver.cc:374] Webserver started at http://127.13.53.1:38489/ using document root and password file I0913 23:13:46.530241 14225 heartbeater.cc:322] Connected to a master server at 127.13.53.62:36199 I0913 23:13:46.530508 14225 heartbeater.cc:406] Registering TS with master... I0913 23:13:46.530880 14225 heartbeater.cc:455] Master 127.13.53.62:36199 requested a full tablet report, sending... I0913 23:13:46.532414 14118 ts_manager.cc:161] Registered new tserver with Master: aec0bff686fa48748eec67f1005139ac (127.13.53.1:34579) I0913 23:13:46.532703 13524 internal_mini_cluster.cc:328] 1 TS(s) registered with all masters after 0.018243323s I0913 23:13:46.534201 14118 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44712 I0913 23:13:46.550000 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:13:46.592088 13524 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0913 23:13:46.606839 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:13:46.607000 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:13:46.607254 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:13:46.622493 13524 tablet_server.cc:87] deduplicated master addresses: 127.13.53.62:36199 I0913 23:13:46.622994 13524 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 18200us I0913 23:13:46.623281 13524 fs_manager.cc:263] Metadata directory not provided I0913 23:13:46.623383 13524 fs_manager.cc:266] Using existing metadata directory in first data directory I0913 23:13:46.625674 13524 fs_manager.cc:399] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s I0913 23:13:46.626575 13524 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0913 23:13:46.632598 13524 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0913 23:13:46.632748 13524 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568416417781642-13524/minicluster-data/ts-0-root uuid: "aec0bff686fa48748eec67f1005139ac" format_stamp: "Formatted at 2019-09-13 23:13:46 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:46.632961 13524 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568416417781642-13524/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568416417781642-13524/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568416417781642-13524/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0913 23:13:46.681751 13524 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0913 23:13:46.681901 13524 ts_tablet_manager.cc:403] Registered 0 tablets I0913 23:13:46.716794 13524 rpc_server.cc:205] RPC server started. Bound to: 127.13.53.1:34579 I0913 23:13:46.718477 13524 webserver.cc:374] Webserver started at http://127.13.53.1:38489/ using document root and password file I0913 23:13:46.723474 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:13:46.766916 14299 heartbeater.cc:322] Connected to a master server at 127.13.53.62:36199 I0913 23:13:46.767360 14299 heartbeater.cc:406] Registering TS with master... I0913 23:13:46.767874 14299 heartbeater.cc:455] Master 127.13.53.62:36199 requested a full tablet report, sending... I0913 23:13:46.770674 14118 ts_manager.cc:161] Re-registered known tserver with Master: aec0bff686fa48748eec67f1005139ac (127.13.53.1:34579) I0913 23:13:46.772312 14118 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44716 I0913 23:13:46.774325 13524 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0913 23:13:46.788815 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:13:46.788997 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:13:46.789294 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:13:46.794432 13524 master.cc:247] Master@127.13.53.62:36199 shutting down... I0913 23:13:46.807202 13524 raft_consensus.cc:2147] T 00000000000000000000000000000000 P d3e419641886423098d5a67c269bee5f [term 1 LEADER]: Raft consensus shutting down. I0913 23:13:46.807575 13524 raft_consensus.cc:2174] T 00000000000000000000000000000000 P d3e419641886423098d5a67c269bee5f [term 1 FOLLOWER]: Raft consensus is shut down! I0913 23:13:46.808043 13524 tablet_replica.cc:273] T 00000000000000000000000000000000 P d3e419641886423098d5a67c269bee5f: stopping tablet replica I0913 23:13:46.816913 13524 master.cc:258] Master@127.13.53.62:36199 shutdown complete. [ OK ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters (669 ms) [ RUN ] TsTabletManagerITest.TestTableStats I0913 23:13:46.832918 13524 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.13.53.62:40439,127.13.53.61:35977,127.13.53.60:42667 I0913 23:13:46.843581 13524 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 18200us I0913 23:13:46.843904 13524 fs_manager.cc:263] Metadata directory not provided I0913 23:13:46.844038 13524 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0913 23:13:46.844214 13524 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0913 23:13:46.844318 13524 server_base.cc:449] Attempting to create new FS layout instead I0913 23:13:46.845192 13524 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/master-0-root/instance: uuid: "46574f27d20e427381df2ea44498c90b" format_stamp: "Formatted at 2019-09-13 23:13:46 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:46.852625 13524 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s I0913 23:13:46.853931 13524 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0913 23:13:46.859956 13524 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0913 23:13:46.860124 13524 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/master-0-root uuid: "46574f27d20e427381df2ea44498c90b" format_stamp: "Formatted at 2019-09-13 23:13:46 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:46.860355 13524 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/master-0-root metadata directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0913 23:13:46.927639 13524 rpc_server.cc:205] RPC server started. Bound to: 127.13.53.62:40439 I0913 23:13:46.929375 13524 webserver.cc:374] Webserver started at http://127.13.53.62:33569/ using document root and password file I0913 23:13:46.931094 14360 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0913 23:13:46.935307 14360 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } has no permanent_uuid. Determining permanent_uuid... I0913 23:13:46.941357 13524 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 18200us I0913 23:13:46.941706 13524 fs_manager.cc:263] Metadata directory not provided I0913 23:13:46.941848 13524 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0913 23:13:46.942051 13524 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0913 23:13:46.942224 13524 server_base.cc:449] Attempting to create new FS layout instead I0913 23:13:46.943362 13524 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/master-1-root/instance: uuid: "b03280a8df7e48f9939e076d6d0138ca" format_stamp: "Formatted at 2019-09-13 23:13:46 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:46.953686 13524 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.011s sys 0.000s I0913 23:13:46.953711 14360 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } has no permanent_uuid. Determining permanent_uuid... I0913 23:13:46.955273 13524 file_cache.cc:465] Constructed file cache lbm with capacity 419430 W0913 23:13:46.955603 14360 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.13.53.61:35977: Network error: Client connection negotiation failed: client connection to 127.13.53.61:35977: connect: Connection refused (error 111) I0913 23:13:46.961616 13524 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0913 23:13:46.961788 13524 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/master-1-root uuid: "b03280a8df7e48f9939e076d6d0138ca" format_stamp: "Formatted at 2019-09-13 23:13:46 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:46.962040 13524 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/master-1-root metadata directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/master-1-root 1 data directories: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/master-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0913 23:13:47.008666 13524 rpc_server.cc:205] RPC server started. Bound to: 127.13.53.61:35977 I0913 23:13:47.010344 13524 webserver.cc:374] Webserver started at http://127.13.53.61:44137/ using document root and password file I0913 23:13:47.011835 14417 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0913 23:13:47.012055 14360 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } attempt: 1 I0913 23:13:47.017529 14417 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } has no permanent_uuid. Determining permanent_uuid... I0913 23:13:47.025830 14360 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } has no permanent_uuid. Determining permanent_uuid... W0913 23:13:47.027483 14360 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.13.53.60:42667: Network error: Client connection negotiation failed: client connection to 127.13.53.60:42667: connect: Connection refused (error 111) I0913 23:13:47.030169 13524 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 18700us I0913 23:13:47.030509 13524 fs_manager.cc:263] Metadata directory not provided I0913 23:13:47.030701 13524 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0913 23:13:47.030956 13524 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0913 23:13:47.031064 13524 server_base.cc:449] Attempting to create new FS layout instead I0913 23:13:47.031826 14417 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } has no permanent_uuid. Determining permanent_uuid... I0913 23:13:47.032033 13524 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/master-2-root/instance: uuid: "fab3d6e5bcad48acbb0fe6b928638607" format_stamp: "Formatted at 2019-09-13 23:13:47 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:47.041802 14417 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } has no permanent_uuid. Determining permanent_uuid... I0913 23:13:47.043138 13524 fs_manager.cc:503] Time spent creating directory manager: real 0.011s user 0.012s sys 0.000s W0913 23:13:47.043422 14417 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.13.53.60:42667: Network error: Client connection negotiation failed: client connection to 127.13.53.60:42667: connect: Connection refused (error 111) I0913 23:13:47.044538 13524 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0913 23:13:47.051352 13524 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0913 23:13:47.051534 13524 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/master-2-root uuid: "fab3d6e5bcad48acbb0fe6b928638607" format_stamp: "Formatted at 2019-09-13 23:13:47 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:47.051757 13524 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/master-2-root metadata directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/master-2-root 1 data directories: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/master-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0913 23:13:47.062824 14360 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } attempt: 1 W0913 23:13:47.064723 14360 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.13.53.60:42667: Network error: Client connection negotiation failed: client connection to 127.13.53.60:42667: connect: Connection refused (error 111) I0913 23:13:47.089807 14417 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } attempt: 1 W0913 23:13:47.091677 14417 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.13.53.60:42667: Network error: Client connection negotiation failed: client connection to 127.13.53.60:42667: connect: Connection refused (error 111) I0913 23:13:47.131078 14360 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } attempt: 2 W0913 23:13:47.133065 14360 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.13.53.60:42667: Network error: Client connection negotiation failed: client connection to 127.13.53.60:42667: connect: Connection refused (error 111) I0913 23:13:47.144309 13524 rpc_server.cc:205] RPC server started. Bound to: 127.13.53.60:42667 I0913 23:13:47.146087 13524 webserver.cc:374] Webserver started at http://127.13.53.60:35131/ using document root and password file I0913 23:13:47.147660 14474 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0913 23:13:47.150174 13524 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0913 23:13:47.152262 14474 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } has no permanent_uuid. Determining permanent_uuid... I0913 23:13:47.159003 14417 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } attempt: 2 I0913 23:13:47.163516 14474 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } has no permanent_uuid. Determining permanent_uuid... I0913 23:13:47.173125 14474 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } has no permanent_uuid. Determining permanent_uuid... I0913 23:13:47.175479 14417 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca: Bootstrap starting. I0913 23:13:47.178203 14417 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca: No blocks or log segments found. Creating new log. I0913 23:13:47.181399 14417 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca: No bootstrap required, opened a new log I0913 23:13:47.183241 14417 raft_consensus.cc:340] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:47.183738 14417 raft_consensus.cc:366] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0913 23:13:47.183915 14417 raft_consensus.cc:705] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03280a8df7e48f9939e076d6d0138ca, State: Initialized, Role: FOLLOWER I0913 23:13:47.184340 14417 consensus_queue.cc:229] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:47.187235 14474 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607: Bootstrap starting. I0913 23:13:47.189437 14474 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607: No blocks or log segments found. Creating new log. I0913 23:13:47.191309 14417 sys_catalog.cc:340] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:47.191413 14478 sys_catalog.cc:340] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:47.191861 14417 sys_catalog.cc:343] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [sys.catalog]: This master's current role is: FOLLOWER I0913 23:13:47.191995 14478 sys_catalog.cc:343] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [sys.catalog]: This master's current role is: FOLLOWER I0913 23:13:47.192780 14417 sys_catalog.cc:428] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [sys.catalog]: configured and running, proceeding with master startup. I0913 23:13:47.193895 14474 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607: No bootstrap required, opened a new log I0913 23:13:47.196137 14474 raft_consensus.cc:340] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:47.196790 14474 raft_consensus.cc:366] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0913 23:13:47.196997 14474 raft_consensus.cc:705] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fab3d6e5bcad48acbb0fe6b928638607, State: Initialized, Role: FOLLOWER W0913 23:13:47.197055 14480 catalog_manager.cc:1129] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0913 23:13:47.197333 14480 catalog_manager.cc:612] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0913 23:13:47.197546 14474 consensus_queue.cc:229] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:47.199057 14481 sys_catalog.cc:340] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:47.199529 14481 sys_catalog.cc:343] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: This master's current role is: FOLLOWER I0913 23:13:47.199852 14474 sys_catalog.cc:340] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:47.200270 14474 sys_catalog.cc:343] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: This master's current role is: FOLLOWER I0913 23:13:47.200944 14474 sys_catalog.cc:428] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: configured and running, proceeding with master startup. I0913 23:13:47.201783 14482 catalog_manager.cc:554] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting I0913 23:13:47.218997 14478 raft_consensus.cc:472] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0913 23:13:47.219398 14478 raft_consensus.cc:494] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:47.220521 14478 leader_election.cc:296] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46574f27d20e427381df2ea44498c90b (127.13.53.62:40439), fab3d6e5bcad48acbb0fe6b928638607 (127.13.53.60:42667) I0913 23:13:47.221388 14451 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b03280a8df7e48f9939e076d6d0138ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fab3d6e5bcad48acbb0fe6b928638607" is_pre_election: true I0913 23:13:47.221963 14451 raft_consensus.cc:2350] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b03280a8df7e48f9939e076d6d0138ca in term 0. I0913 23:13:47.222785 14368 leader_election.cc:310] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b03280a8df7e48f9939e076d6d0138ca, fab3d6e5bcad48acbb0fe6b928638607; no voters: I0913 23:13:47.223378 14478 raft_consensus.cc:2644] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 0 FOLLOWER]: Leader pre-election won for term 1 I0913 23:13:47.223577 14478 raft_consensus.cc:472] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0913 23:13:47.223790 14478 raft_consensus.cc:2890] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 0 FOLLOWER]: Advancing to term 1 I0913 23:13:47.228899 14478 raft_consensus.cc:494] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:47.230388 14478 leader_election.cc:296] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [CANDIDATE]: Term 1 election: Requested vote from peers 46574f27d20e427381df2ea44498c90b (127.13.53.62:40439), fab3d6e5bcad48acbb0fe6b928638607 (127.13.53.60:42667) I0913 23:13:47.231122 14451 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b03280a8df7e48f9939e076d6d0138ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fab3d6e5bcad48acbb0fe6b928638607" I0913 23:13:47.231528 14451 raft_consensus.cc:2890] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 0 FOLLOWER]: Advancing to term 1 I0913 23:13:47.234972 14451 raft_consensus.cc:2350] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b03280a8df7e48f9939e076d6d0138ca in term 1. I0913 23:13:47.235839 14368 leader_election.cc:310] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b03280a8df7e48f9939e076d6d0138ca, fab3d6e5bcad48acbb0fe6b928638607; no voters: I0913 23:13:47.236584 14478 raft_consensus.cc:2644] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 1 FOLLOWER]: Leader election won for term 1 I0913 23:13:47.237869 14478 raft_consensus.cc:667] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 1 LEADER]: Becoming Leader. State: Replica: b03280a8df7e48f9939e076d6d0138ca, State: Running, Role: LEADER I0913 23:13:47.238453 14360 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } attempt: 3 I0913 23:13:47.238559 14478 consensus_queue.cc:206] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:47.242197 14483 sys_catalog.cc:340] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader b03280a8df7e48f9939e076d6d0138ca. Latest consensus state: current_term: 1 leader_uuid: "b03280a8df7e48f9939e076d6d0138ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:47.243157 14483 sys_catalog.cc:343] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [sys.catalog]: This master's current role is: LEADER I0913 23:13:47.244460 14485 catalog_manager.cc:1071] Loading table and tablet metadata into memory... I0913 23:13:47.247718 14485 catalog_manager.cc:1082] Initializing Kudu internal certificate authority... I0913 23:13:47.254931 14360 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b: Bootstrap starting. I0913 23:13:47.257817 14360 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b: No blocks or log segments found. Creating new log. I0913 23:13:47.263986 14360 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b: No bootstrap required, opened a new log I0913 23:13:47.265307 14360 raft_consensus.cc:340] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:47.265799 14360 raft_consensus.cc:366] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0913 23:13:47.265965 14360 raft_consensus.cc:705] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46574f27d20e427381df2ea44498c90b, State: Initialized, Role: FOLLOWER I0913 23:13:47.266501 14360 consensus_queue.cc:229] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:47.270645 14360 sys_catalog.cc:340] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:47.271132 14360 sys_catalog.cc:343] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: This master's current role is: FOLLOWER I0913 23:13:47.270872 14487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:47.271598 14487 sys_catalog.cc:343] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: This master's current role is: FOLLOWER I0913 23:13:47.271942 14360 sys_catalog.cc:428] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: configured and running, proceeding with master startup. I0913 23:13:47.271996 14336 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b03280a8df7e48f9939e076d6d0138ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46574f27d20e427381df2ea44498c90b" I0913 23:13:47.272327 14337 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b03280a8df7e48f9939e076d6d0138ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46574f27d20e427381df2ea44498c90b" is_pre_election: true I0913 23:13:47.272842 14336 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 0 FOLLOWER]: Advancing to term 1 I0913 23:13:47.279111 14336 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b03280a8df7e48f9939e076d6d0138ca in term 1. I0913 23:13:47.280232 14337 raft_consensus.cc:2325] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b03280a8df7e48f9939e076d6d0138ca for term 1 because replica is already servicing an update from a current leader or another vote. I0913 23:13:47.282527 13524 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 1 I0913 23:13:47.282728 13524 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 2 W0913 23:13:47.284636 14488 catalog_manager.cc:1129] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0913 23:13:47.284889 14488 catalog_manager.cc:612] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0913 23:13:47.296353 13524 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 18700us I0913 23:13:47.296722 13524 fs_manager.cc:263] Metadata directory not provided I0913 23:13:47.296895 13524 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0913 23:13:47.297094 13524 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0913 23:13:47.297204 13524 server_base.cc:449] Attempting to create new FS layout instead I0913 23:13:47.298189 13524 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/ts-0-root/instance: uuid: "93a20b201a904b628926e173bbf23463" format_stamp: "Formatted at 2019-09-13 23:13:47 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:47.307679 13524 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s I0913 23:13:47.309291 13524 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0913 23:13:47.310567 14337 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 1 FOLLOWER]: Refusing update from remote peer b03280a8df7e48f9939e076d6d0138ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0913 23:13:47.310600 14451 raft_consensus.cc:1184] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 1 FOLLOWER]: Refusing update from remote peer b03280a8df7e48f9939e076d6d0138ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0913 23:13:47.311772 14478 consensus_queue.cc:984] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0913 23:13:47.312273 14483 consensus_queue.cc:984] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0913 23:13:47.318706 13524 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0913 23:13:47.319047 13524 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/ts-0-root uuid: "93a20b201a904b628926e173bbf23463" format_stamp: "Formatted at 2019-09-13 23:13:47 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:47.319418 13524 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0913 23:13:47.322185 14481 sys_catalog.cc:340] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b03280a8df7e48f9939e076d6d0138ca. Latest consensus state: current_term: 1 leader_uuid: "b03280a8df7e48f9939e076d6d0138ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:47.323066 14481 sys_catalog.cc:343] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: This master's current role is: FOLLOWER I0913 23:13:47.324681 14487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b03280a8df7e48f9939e076d6d0138ca. Latest consensus state: current_term: 1 leader_uuid: "b03280a8df7e48f9939e076d6d0138ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:47.325407 14487 sys_catalog.cc:343] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: This master's current role is: FOLLOWER I0913 23:13:47.326846 14490 mvcc.cc:203] Tried to move safe_time back from 6424233686253481984 to 6424233685975818240. Current Snapshot: MvccSnapshot[committed={T|T < 6424233686253481984}] I0913 23:13:47.336338 14483 sys_catalog.cc:340] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b03280a8df7e48f9939e076d6d0138ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:47.337090 14483 sys_catalog.cc:343] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [sys.catalog]: This master's current role is: LEADER I0913 23:13:47.339521 14483 sys_catalog.cc:340] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b03280a8df7e48f9939e076d6d0138ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:47.340376 14483 sys_catalog.cc:343] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [sys.catalog]: This master's current role is: LEADER I0913 23:13:47.340694 14485 catalog_manager.cc:966] Generated new certificate authority record I0913 23:13:47.341711 14487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b03280a8df7e48f9939e076d6d0138ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:47.342468 14487 sys_catalog.cc:343] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: This master's current role is: FOLLOWER I0913 23:13:47.342990 14485 catalog_manager.cc:1091] Loading token signing keys... I0913 23:13:47.346387 14481 sys_catalog.cc:340] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b03280a8df7e48f9939e076d6d0138ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:47.347082 14481 sys_catalog.cc:343] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: This master's current role is: FOLLOWER I0913 23:13:47.370553 14485 catalog_manager.cc:4394] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca: Generated new TSK 0 I0913 23:13:47.383915 13524 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0913 23:13:47.384094 13524 ts_tablet_manager.cc:403] Registered 0 tablets I0913 23:13:47.418488 13524 rpc_server.cc:205] RPC server started. Bound to: 127.13.53.1:44785 I0913 23:13:47.420264 13524 webserver.cc:374] Webserver started at http://127.13.53.1:40543/ using document root and password file I0913 23:13:47.444479 14563 heartbeater.cc:322] Connected to a master server at 127.13.53.61:35977 I0913 23:13:47.444803 14563 heartbeater.cc:406] Registering TS with master... I0913 23:13:47.445257 13524 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 18700us I0913 23:13:47.445752 13524 fs_manager.cc:263] Metadata directory not provided I0913 23:13:47.445806 14563 heartbeater.cc:455] Master 127.13.53.61:35977 requested a full tablet report, sending... I0913 23:13:47.445966 13524 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0913 23:13:47.446389 13524 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0913 23:13:47.446553 13524 server_base.cc:449] Attempting to create new FS layout instead I0913 23:13:47.447685 13524 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/ts-1-root/instance: uuid: "0cad5c4a245f4b3dbd66bb7259f45090" format_stamp: "Formatted at 2019-09-13 23:13:47 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:47.447933 14562 heartbeater.cc:322] Connected to a master server at 127.13.53.60:42667 I0913 23:13:47.448221 14564 heartbeater.cc:322] Connected to a master server at 127.13.53.62:40439 I0913 23:13:47.448308 14562 heartbeater.cc:406] Registering TS with master... I0913 23:13:47.448550 14564 heartbeater.cc:406] Registering TS with master... I0913 23:13:47.448845 14562 heartbeater.cc:455] Master 127.13.53.60:42667 requested a full tablet report, sending... I0913 23:13:47.447875 14384 ts_manager.cc:161] Registered new tserver with Master: 93a20b201a904b628926e173bbf23463 (127.13.53.1:44785) I0913 23:13:47.451699 14384 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38016 I0913 23:13:47.448930 14564 heartbeater.cc:455] Master 127.13.53.62:40439 requested a full tablet report, sending... I0913 23:13:47.454607 14441 ts_manager.cc:161] Registered new tserver with Master: 93a20b201a904b628926e173bbf23463 (127.13.53.1:44785) I0913 23:13:47.455240 14327 ts_manager.cc:161] Registered new tserver with Master: 93a20b201a904b628926e173bbf23463 (127.13.53.1:44785) I0913 23:13:47.459650 13524 fs_manager.cc:503] Time spent creating directory manager: real 0.011s user 0.008s sys 0.004s I0913 23:13:47.461022 13524 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0913 23:13:47.468533 13524 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0913 23:13:47.468688 13524 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/ts-1-root uuid: "0cad5c4a245f4b3dbd66bb7259f45090" format_stamp: "Formatted at 2019-09-13 23:13:47 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:47.468883 13524 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0913 23:13:47.507637 13524 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0913 23:13:47.507802 13524 ts_tablet_manager.cc:403] Registered 0 tablets I0913 23:13:47.555269 14563 heartbeater.cc:447] Master 127.13.53.61:35977 was elected leader, sending a full tablet report... I0913 23:13:47.573073 13524 rpc_server.cc:205] RPC server started. Bound to: 127.13.53.2:41231 I0913 23:13:47.575314 13524 webserver.cc:374] Webserver started at http://127.13.53.2:33761/ using document root and password file I0913 23:13:47.597033 14633 heartbeater.cc:322] Connected to a master server at 127.13.53.60:42667 I0913 23:13:47.597393 14633 heartbeater.cc:406] Registering TS with master... I0913 23:13:47.598165 14634 heartbeater.cc:322] Connected to a master server at 127.13.53.61:35977 I0913 23:13:47.598495 14634 heartbeater.cc:406] Registering TS with master... I0913 23:13:47.598985 14634 heartbeater.cc:455] Master 127.13.53.61:35977 requested a full tablet report, sending... I0913 23:13:47.600316 14636 heartbeater.cc:322] Connected to a master server at 127.13.53.62:40439 I0913 23:13:47.600600 14636 heartbeater.cc:406] Registering TS with master... I0913 23:13:47.600661 14384 ts_manager.cc:161] Registered new tserver with Master: 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231) I0913 23:13:47.601095 14636 heartbeater.cc:455] Master 127.13.53.62:40439 requested a full tablet report, sending... I0913 23:13:47.602490 14384 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38022 I0913 23:13:47.603158 14327 ts_manager.cc:161] Registered new tserver with Master: 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231) I0913 23:13:47.598031 14633 heartbeater.cc:455] Master 127.13.53.60:42667 requested a full tablet report, sending... I0913 23:13:47.605805 14441 ts_manager.cc:161] Registered new tserver with Master: 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231) I0913 23:13:47.608311 13524 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 18700us I0913 23:13:47.608613 13524 fs_manager.cc:263] Metadata directory not provided I0913 23:13:47.608733 13524 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0913 23:13:47.608883 13524 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0913 23:13:47.608989 13524 server_base.cc:449] Attempting to create new FS layout instead I0913 23:13:47.609774 13524 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/ts-2-root/instance: uuid: "a66db810508f4e35bac62f5c6c9f6e9c" format_stamp: "Formatted at 2019-09-13 23:13:47 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:47.619854 13524 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.005s sys 0.005s I0913 23:13:47.621273 13524 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0913 23:13:47.629591 13524 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0913 23:13:47.629778 13524 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/ts-2-root uuid: "a66db810508f4e35bac62f5c6c9f6e9c" format_stamp: "Formatted at 2019-09-13 23:13:47 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:47.630053 13524 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0913 23:13:47.678769 13524 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0913 23:13:47.678982 13524 ts_tablet_manager.cc:403] Registered 0 tablets I0913 23:13:47.705180 14634 heartbeater.cc:447] Master 127.13.53.61:35977 was elected leader, sending a full tablet report... I0913 23:13:47.746963 13524 rpc_server.cc:205] RPC server started. Bound to: 127.13.53.3:37091 I0913 23:13:47.749338 13524 webserver.cc:374] Webserver started at http://127.13.53.3:44775/ using document root and password file I0913 23:13:47.779196 14704 heartbeater.cc:322] Connected to a master server at 127.13.53.60:42667 I0913 23:13:47.779198 14705 heartbeater.cc:322] Connected to a master server at 127.13.53.61:35977 I0913 23:13:47.779506 14704 heartbeater.cc:406] Registering TS with master... I0913 23:13:47.779793 14705 heartbeater.cc:406] Registering TS with master... I0913 23:13:47.780086 14704 heartbeater.cc:455] Master 127.13.53.60:42667 requested a full tablet report, sending... I0913 23:13:47.780648 14705 heartbeater.cc:455] Master 127.13.53.61:35977 requested a full tablet report, sending... I0913 23:13:47.782094 14384 ts_manager.cc:161] Registered new tserver with Master: a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3:37091) I0913 23:13:47.782303 14441 ts_manager.cc:161] Registered new tserver with Master: a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3:37091) I0913 23:13:47.784138 14384 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38028 I0913 23:13:47.784627 14707 heartbeater.cc:322] Connected to a master server at 127.13.53.62:40439 I0913 23:13:47.784953 14707 heartbeater.cc:406] Registering TS with master... I0913 23:13:47.785408 14707 heartbeater.cc:455] Master 127.13.53.62:40439 requested a full tablet report, sending... I0913 23:13:47.787518 14327 ts_manager.cc:161] Registered new tserver with Master: a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3:37091) I0913 23:13:47.788082 13524 internal_mini_cluster.cc:328] 3 TS(s) registered with all masters after 0.030403609s I0913 23:13:47.828465 14384 catalog_manager.cc:1425] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38034: name: "test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } W0913 23:13:47.829476 14384 catalog_manager.cc:1590] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0913 23:13:47.887073 14705 heartbeater.cc:447] Master 127.13.53.61:35977 was elected leader, sending a full tablet report... I0913 23:13:47.948454 14529 tablet_service.cc:1022] Processing CreateTablet for tablet bf89c16ce260470a863711ddbbdb3d68 (table=test-table [id=334306099b944525a47cb78bb2cafd10]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0913 23:13:47.947906 14527 tablet_service.cc:1022] Processing CreateTablet for tablet 82f65eba7703468ca650e7ca50d39549 (table=test-table [id=334306099b944525a47cb78bb2cafd10]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0913 23:13:47.949542 14528 tablet_service.cc:1022] Processing CreateTablet for tablet 8ad9594f28c64b8ba3dacd888b84312b (table=test-table [id=334306099b944525a47cb78bb2cafd10]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0913 23:13:47.950626 14527 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 82f65eba7703468ca650e7ca50d39549. 1 dirs total, 0 dirs full, 0 dirs failed I0913 23:13:47.951526 14528 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 8ad9594f28c64b8ba3dacd888b84312b. 1 dirs total, 0 dirs full, 0 dirs failed I0913 23:13:47.955725 14529 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet bf89c16ce260470a863711ddbbdb3d68. 1 dirs total, 0 dirs full, 0 dirs failed I0913 23:13:47.960220 14600 tablet_service.cc:1022] Processing CreateTablet for tablet bf89c16ce260470a863711ddbbdb3d68 (table=test-table [id=334306099b944525a47cb78bb2cafd10]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0913 23:13:47.961088 14598 tablet_service.cc:1022] Processing CreateTablet for tablet 82f65eba7703468ca650e7ca50d39549 (table=test-table [id=334306099b944525a47cb78bb2cafd10]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0913 23:13:47.962766 14600 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet bf89c16ce260470a863711ddbbdb3d68. 1 dirs total, 0 dirs full, 0 dirs failed I0913 23:13:47.964160 14598 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 82f65eba7703468ca650e7ca50d39549. 1 dirs total, 0 dirs full, 0 dirs failed I0913 23:13:47.963881 14599 tablet_service.cc:1022] Processing CreateTablet for tablet 8ad9594f28c64b8ba3dacd888b84312b (table=test-table [id=334306099b944525a47cb78bb2cafd10]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0913 23:13:47.965555 14599 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 8ad9594f28c64b8ba3dacd888b84312b. 1 dirs total, 0 dirs full, 0 dirs failed I0913 23:13:47.987748 14669 tablet_service.cc:1022] Processing CreateTablet for tablet 82f65eba7703468ca650e7ca50d39549 (table=test-table [id=334306099b944525a47cb78bb2cafd10]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0913 23:13:47.989832 14669 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 82f65eba7703468ca650e7ca50d39549. 1 dirs total, 0 dirs full, 0 dirs failed I0913 23:13:47.991251 14670 tablet_service.cc:1022] Processing CreateTablet for tablet 8ad9594f28c64b8ba3dacd888b84312b (table=test-table [id=334306099b944525a47cb78bb2cafd10]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0913 23:13:47.994295 14670 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 8ad9594f28c64b8ba3dacd888b84312b. 1 dirs total, 0 dirs full, 0 dirs failed I0913 23:13:47.993674 14671 tablet_service.cc:1022] Processing CreateTablet for tablet bf89c16ce260470a863711ddbbdb3d68 (table=test-table [id=334306099b944525a47cb78bb2cafd10]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0913 23:13:47.995956 14671 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet bf89c16ce260470a863711ddbbdb3d68. 1 dirs total, 0 dirs full, 0 dirs failed I0913 23:13:47.996371 14726 tablet_bootstrap.cc:439] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463: Bootstrap starting. I0913 23:13:48.006176 14727 tablet_bootstrap.cc:439] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090: Bootstrap starting. I0913 23:13:48.009739 14726 tablet_bootstrap.cc:590] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463: No blocks or log segments found. Creating new log. I0913 23:13:48.010767 14728 tablet_bootstrap.cc:439] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c: Bootstrap starting. I0913 23:13:48.014292 14727 tablet_bootstrap.cc:590] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090: No blocks or log segments found. Creating new log. I0913 23:13:48.014686 14728 tablet_bootstrap.cc:590] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c: No blocks or log segments found. Creating new log. I0913 23:13:48.014849 14726 tablet_bootstrap.cc:439] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463: No bootstrap required, opened a new log I0913 23:13:48.015311 14726 ts_tablet_manager.cc:1085] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.009s I0913 23:13:48.020452 14727 tablet_bootstrap.cc:439] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090: No bootstrap required, opened a new log I0913 23:13:48.020970 14727 ts_tablet_manager.cc:1085] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.005s I0913 23:13:48.021153 14726 raft_consensus.cc:340] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:48.021978 14726 raft_consensus.cc:366] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0913 23:13:48.022260 14726 raft_consensus.cc:705] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93a20b201a904b628926e173bbf23463, State: Initialized, Role: FOLLOWER I0913 23:13:48.022845 14726 consensus_queue.cc:229] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:48.023507 14728 tablet_bootstrap.cc:439] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c: No bootstrap required, opened a new log I0913 23:13:48.027889 14728 ts_tablet_manager.cc:1085] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.007s I0913 23:13:48.023859 14727 raft_consensus.cc:340] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:48.028410 14727 raft_consensus.cc:366] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0913 23:13:48.028694 14727 raft_consensus.cc:705] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cad5c4a245f4b3dbd66bb7259f45090, State: Initialized, Role: FOLLOWER I0913 23:13:48.029255 14727 consensus_queue.cc:229] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:48.034312 14728 raft_consensus.cc:340] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:48.035033 14728 raft_consensus.cc:366] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0913 23:13:48.035239 14728 raft_consensus.cc:705] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a66db810508f4e35bac62f5c6c9f6e9c, State: Initialized, Role: FOLLOWER I0913 23:13:48.035722 14728 consensus_queue.cc:229] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:48.041415 14726 ts_tablet_manager.cc:1113] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463: Time spent starting tablet: real 0.026s user 0.004s sys 0.005s I0913 23:13:48.041415 14727 ts_tablet_manager.cc:1113] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090: Time spent starting tablet: real 0.020s user 0.007s sys 0.001s I0913 23:13:48.049808 14726 tablet_bootstrap.cc:439] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463: Bootstrap starting. I0913 23:13:48.051738 14727 tablet_bootstrap.cc:439] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090: Bootstrap starting. I0913 23:13:48.065040 14727 tablet_bootstrap.cc:590] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090: No blocks or log segments found. Creating new log. I0913 23:13:48.067634 14726 tablet_bootstrap.cc:590] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463: No blocks or log segments found. Creating new log. I0913 23:13:48.069145 14733 raft_consensus.cc:472] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0913 23:13:48.069595 14728 ts_tablet_manager.cc:1113] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c: Time spent starting tablet: real 0.041s user 0.005s sys 0.009s I0913 23:13:48.070446 14728 tablet_bootstrap.cc:439] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c: Bootstrap starting. I0913 23:13:48.069658 14733 raft_consensus.cc:494] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:48.073977 14733 leader_election.cc:296] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93a20b201a904b628926e173bbf23463 (127.13.53.1:44785), a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3:37091) I0913 23:13:48.095780 14732 raft_consensus.cc:472] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0913 23:13:48.097189 14732 raft_consensus.cc:494] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:48.119097 14732 leader_election.cc:296] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231), a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3:37091) I0913 23:13:48.123793 14539 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "82f65eba7703468ca650e7ca50d39549" candidate_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93a20b201a904b628926e173bbf23463" is_pre_election: true I0913 23:13:48.124399 14734 raft_consensus.cc:472] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0913 23:13:48.124905 14734 raft_consensus.cc:494] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } W0913 23:13:48.126339 14574 leader_election.cc:348] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93a20b201a904b628926e173bbf23463 (127.13.53.1:44785): Illegal state: must be running to vote when last-logged opid is not known I0913 23:13:48.152828 14727 tablet_bootstrap.cc:439] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090: No bootstrap required, opened a new log I0913 23:13:48.147810 14728 tablet_bootstrap.cc:590] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c: No blocks or log segments found. Creating new log. I0913 23:13:48.164384 14727 ts_tablet_manager.cc:1085] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090: Time spent bootstrapping tablet: real 0.113s user 0.007s sys 0.024s I0913 23:13:48.167426 14727 raft_consensus.cc:340] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:48.168220 14727 raft_consensus.cc:366] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0913 23:13:48.168462 14727 raft_consensus.cc:705] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cad5c4a245f4b3dbd66bb7259f45090, State: Initialized, Role: FOLLOWER I0913 23:13:48.181593 14539 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "82f65eba7703468ca650e7ca50d39549" candidate_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93a20b201a904b628926e173bbf23463" is_pre_election: true I0913 23:13:48.151957 14734 leader_election.cc:296] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93a20b201a904b628926e173bbf23463 (127.13.53.1:44785), 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231) I0913 23:13:48.178609 14727 consensus_queue.cc:229] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:48.182157 14726 tablet_bootstrap.cc:439] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463: No bootstrap required, opened a new log I0913 23:13:48.184402 14726 ts_tablet_manager.cc:1085] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463: Time spent bootstrapping tablet: real 0.135s user 0.014s sys 0.002s I0913 23:13:48.184815 14610 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8ad9594f28c64b8ba3dacd888b84312b" candidate_uuid: "93a20b201a904b628926e173bbf23463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" is_pre_election: true I0913 23:13:48.185596 14610 raft_consensus.cc:2350] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93a20b201a904b628926e173bbf23463 in term 0. W0913 23:13:48.192003 14645 leader_election.cc:348] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 93a20b201a904b628926e173bbf23463 (127.13.53.1:44785): Illegal state: must be running to vote when last-logged opid is not known I0913 23:13:48.195235 14497 leader_election.cc:310] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cad5c4a245f4b3dbd66bb7259f45090, 93a20b201a904b628926e173bbf23463; no voters: I0913 23:13:48.196600 14732 raft_consensus.cc:2644] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0913 23:13:48.196943 14732 raft_consensus.cc:472] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0913 23:13:48.197213 14732 raft_consensus.cc:2890] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [term 0 FOLLOWER]: Advancing to term 1 I0913 23:13:48.221583 14732 raft_consensus.cc:494] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:48.234226 14482 catalog_manager.cc:1127] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607: acquiring CA information for follower catalog manager: success I0913 23:13:48.243957 14610 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8ad9594f28c64b8ba3dacd888b84312b" candidate_uuid: "93a20b201a904b628926e173bbf23463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" I0913 23:13:48.244556 14610 raft_consensus.cc:2890] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [term 0 FOLLOWER]: Advancing to term 1 I0913 23:13:48.245594 14726 raft_consensus.cc:340] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } I0913 23:13:48.246999 14732 leader_election.cc:296] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [CANDIDATE]: Term 1 election: Requested vote from peers 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231), a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3:37091) I0913 23:13:48.247148 14726 raft_consensus.cc:366] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0913 23:13:48.247539 14726 raft_consensus.cc:705] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93a20b201a904b628926e173bbf23463, State: Initialized, Role: FOLLOWER I0913 23:13:48.247916 14609 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "82f65eba7703468ca650e7ca50d39549" candidate_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" is_pre_election: true I0913 23:13:48.248958 14609 raft_consensus.cc:2350] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a66db810508f4e35bac62f5c6c9f6e9c in term 0. I0913 23:13:48.250248 14643 leader_election.cc:310] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cad5c4a245f4b3dbd66bb7259f45090, a66db810508f4e35bac62f5c6c9f6e9c; no voters: 93a20b201a904b628926e173bbf23463 I0913 23:13:48.251355 14734 raft_consensus.cc:2644] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [term 0 FOLLOWER]: Leader pre-election won for term 1 I0913 23:13:48.251981 14734 raft_consensus.cc:472] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0913 23:13:48.252218 14734 raft_consensus.cc:2890] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [term 0 FOLLOWER]: Advancing to term 1 I0913 23:13:48.252297 14727 ts_tablet_manager.cc:1113] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090: Time spent starting tablet: real 0.087s user 0.005s sys 0.000s I0913 23:13:48.253278 14726 consensus_queue.cc:229] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } I0913 23:13:48.256297 14681 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "82f65eba7703468ca650e7ca50d39549" candidate_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" is_pre_election: true I0913 23:13:48.257827 14726 ts_tablet_manager.cc:1113] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463: Time spent starting tablet: real 0.073s user 0.022s sys 0.042s I0913 23:13:48.258400 14610 raft_consensus.cc:2350] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93a20b201a904b628926e173bbf23463 in term 1. I0913 23:13:48.258821 14726 tablet_bootstrap.cc:439] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463: Bootstrap starting. I0913 23:13:48.259552 14727 tablet_bootstrap.cc:439] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090: Bootstrap starting. I0913 23:13:48.259865 14497 leader_election.cc:310] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cad5c4a245f4b3dbd66bb7259f45090, 93a20b201a904b628926e173bbf23463; no voters: I0913 23:13:48.261268 14732 raft_consensus.cc:2644] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [term 1 FOLLOWER]: Leader election won for term 1 I0913 23:13:48.265781 14482 catalog_manager.cc:1155] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0913 23:13:48.273877 14734 raft_consensus.cc:494] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:48.278203 14681 raft_consensus.cc:2280] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0cad5c4a245f4b3dbd66bb7259f45090 in current term 1: Already voted for candidate a66db810508f4e35bac62f5c6c9f6e9c in this term. I0913 23:13:48.282030 14571 leader_election.cc:310] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cad5c4a245f4b3dbd66bb7259f45090; no voters: 93a20b201a904b628926e173bbf23463, a66db810508f4e35bac62f5c6c9f6e9c I0913 23:13:48.283861 14733 raft_consensus.cc:2890] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 0 FOLLOWER]: Advancing to term 1 I0913 23:13:48.288368 14539 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "82f65eba7703468ca650e7ca50d39549" candidate_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93a20b201a904b628926e173bbf23463" W0913 23:13:48.290438 14645 leader_election.cc:348] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 93a20b201a904b628926e173bbf23463 (127.13.53.1:44785): Illegal state: must be running to vote when last-logged opid is not known I0913 23:13:48.295506 14733 raft_consensus.cc:2592] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0913 23:13:48.297042 14488 catalog_manager.cc:1127] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b: acquiring CA information for follower catalog manager: success I0913 23:13:48.298867 14680 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8ad9594f28c64b8ba3dacd888b84312b" candidate_uuid: "93a20b201a904b628926e173bbf23463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" W0913 23:13:48.300710 14496 leader_election.cc:348] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3:37091): Illegal state: must be running to vote when last-logged opid is not known I0913 23:13:48.301602 14681 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8ad9594f28c64b8ba3dacd888b84312b" candidate_uuid: "93a20b201a904b628926e173bbf23463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" is_pre_election: true I0913 23:13:48.301930 14488 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 W0913 23:13:48.303303 14496 leader_election.cc:348] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3:37091): Illegal state: must be running to vote when last-logged opid is not known I0913 23:13:48.308543 14726 tablet_bootstrap.cc:590] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463: No blocks or log segments found. Creating new log. I0913 23:13:48.313983 14734 leader_election.cc:296] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [CANDIDATE]: Term 1 election: Requested vote from peers 93a20b201a904b628926e173bbf23463 (127.13.53.1:44785), 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231) I0913 23:13:48.316499 14728 tablet_bootstrap.cc:439] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c: No bootstrap required, opened a new log I0913 23:13:48.317028 14728 ts_tablet_manager.cc:1085] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c: Time spent bootstrapping tablet: real 0.247s user 0.048s sys 0.065s I0913 23:13:48.315039 14610 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "82f65eba7703468ca650e7ca50d39549" candidate_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" I0913 23:13:48.314251 14732 raft_consensus.cc:667] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [term 1 LEADER]: Becoming Leader. State: Replica: 93a20b201a904b628926e173bbf23463, State: Running, Role: LEADER I0913 23:13:48.319300 14732 consensus_queue.cc:206] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:48.319687 14728 raft_consensus.cc:340] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } I0913 23:13:48.321130 14728 raft_consensus.cc:366] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0913 23:13:48.321416 14728 raft_consensus.cc:705] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a66db810508f4e35bac62f5c6c9f6e9c, State: Initialized, Role: FOLLOWER I0913 23:13:48.322424 14727 tablet_bootstrap.cc:590] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090: No blocks or log segments found. Creating new log. I0913 23:13:48.322135 14728 consensus_queue.cc:229] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } I0913 23:13:48.326299 14728 ts_tablet_manager.cc:1113] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s I0913 23:13:48.325014 14610 raft_consensus.cc:2350] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a66db810508f4e35bac62f5c6c9f6e9c in term 1. I0913 23:13:48.329825 14728 tablet_bootstrap.cc:439] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c: Bootstrap starting. I0913 23:13:48.332887 14643 leader_election.cc:310] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cad5c4a245f4b3dbd66bb7259f45090, a66db810508f4e35bac62f5c6c9f6e9c; no voters: 93a20b201a904b628926e173bbf23463 I0913 23:13:48.333998 14734 raft_consensus.cc:2644] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [term 1 FOLLOWER]: Leader election won for term 1 I0913 23:13:48.338286 14726 tablet_bootstrap.cc:439] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463: No bootstrap required, opened a new log I0913 23:13:48.338901 14734 raft_consensus.cc:667] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [term 1 LEADER]: Becoming Leader. State: Replica: a66db810508f4e35bac62f5c6c9f6e9c, State: Running, Role: LEADER I0913 23:13:48.339092 14726 ts_tablet_manager.cc:1085] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463: Time spent bootstrapping tablet: real 0.080s user 0.007s sys 0.003s I0913 23:13:48.339538 14734 consensus_queue.cc:206] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:48.340909 14382 catalog_manager.cc:4079] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 reported cstate change: term changed from 0 to 1, leader changed from to 93a20b201a904b628926e173bbf23463 (127.13.53.1). New cstate: current_term: 1 leader_uuid: "93a20b201a904b628926e173bbf23463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } health_report { overall_health: UNKNOWN } } } I0913 23:13:48.343029 14732 raft_consensus.cc:472] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0913 23:13:48.343487 14732 raft_consensus.cc:494] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } I0913 23:13:48.345836 14726 raft_consensus.cc:340] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:48.347327 14726 raft_consensus.cc:366] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0913 23:13:48.347573 14726 raft_consensus.cc:705] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93a20b201a904b628926e173bbf23463, State: Initialized, Role: FOLLOWER I0913 23:13:48.347995 14681 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bf89c16ce260470a863711ddbbdb3d68" candidate_uuid: "93a20b201a904b628926e173bbf23463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" is_pre_election: true I0913 23:13:48.348598 14681 raft_consensus.cc:2350] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93a20b201a904b628926e173bbf23463 in term 0. I0913 23:13:48.349684 14496 leader_election.cc:310] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93a20b201a904b628926e173bbf23463, a66db810508f4e35bac62f5c6c9f6e9c; no voters: I0913 23:13:48.351373 14743 raft_consensus.cc:2644] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0913 23:13:48.357595 14743 raft_consensus.cc:472] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0913 23:13:48.357990 14743 raft_consensus.cc:2890] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [term 0 FOLLOWER]: Advancing to term 1 I0913 23:13:48.348287 14726 consensus_queue.cc:229] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:48.353782 14610 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bf89c16ce260470a863711ddbbdb3d68" candidate_uuid: "93a20b201a904b628926e173bbf23463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" is_pre_election: true I0913 23:13:48.352557 14732 leader_election.cc:296] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3:37091), 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231) I0913 23:13:48.361341 14383 catalog_manager.cc:4079] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c reported cstate change: term changed from 0 to 1, leader changed from to a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3). New cstate: current_term: 1 leader_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } health_report { overall_health: HEALTHY } } } W0913 23:13:48.365404 14497 leader_election.cc:348] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231): Illegal state: must be running to vote when last-logged opid is not known I0913 23:13:48.368160 14726 ts_tablet_manager.cc:1113] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s I0913 23:13:48.370816 14743 raft_consensus.cc:494] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } I0913 23:13:48.374322 14681 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bf89c16ce260470a863711ddbbdb3d68" candidate_uuid: "93a20b201a904b628926e173bbf23463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" I0913 23:13:48.375022 14681 raft_consensus.cc:2890] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [term 0 FOLLOWER]: Advancing to term 1 I0913 23:13:48.376984 14610 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bf89c16ce260470a863711ddbbdb3d68" candidate_uuid: "93a20b201a904b628926e173bbf23463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" I0913 23:13:48.378512 14743 leader_election.cc:296] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [CANDIDATE]: Term 1 election: Requested vote from peers a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3:37091), 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231) W0913 23:13:48.379079 14497 leader_election.cc:348] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231): Illegal state: must be running to vote when last-logged opid is not known I0913 23:13:48.381870 14681 raft_consensus.cc:2350] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93a20b201a904b628926e173bbf23463 in term 1. I0913 23:13:48.383450 14496 leader_election.cc:310] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93a20b201a904b628926e173bbf23463, a66db810508f4e35bac62f5c6c9f6e9c; no voters: 0cad5c4a245f4b3dbd66bb7259f45090 I0913 23:13:48.385242 14743 raft_consensus.cc:2644] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [term 1 FOLLOWER]: Leader election won for term 1 I0913 23:13:48.385589 14728 tablet_bootstrap.cc:590] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c: No blocks or log segments found. Creating new log. I0913 23:13:48.385730 14743 raft_consensus.cc:667] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [term 1 LEADER]: Becoming Leader. State: Replica: 93a20b201a904b628926e173bbf23463, State: Running, Role: LEADER I0913 23:13:48.386724 14743 consensus_queue.cc:206] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } I0913 23:13:48.388742 14727 tablet_bootstrap.cc:439] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090: No bootstrap required, opened a new log I0913 23:13:48.389253 14727 ts_tablet_manager.cc:1085] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090: Time spent bootstrapping tablet: real 0.130s user 0.011s sys 0.012s I0913 23:13:48.392145 14727 raft_consensus.cc:340] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } I0913 23:13:48.392913 14727 raft_consensus.cc:366] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0913 23:13:48.393157 14727 raft_consensus.cc:705] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cad5c4a245f4b3dbd66bb7259f45090, State: Initialized, Role: FOLLOWER I0913 23:13:48.393893 14727 consensus_queue.cc:229] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } W0913 23:13:48.404768 14496 consensus_peers.cc:458] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 -> Peer a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3:37091): Couldn't send request to peer a66db810508f4e35bac62f5c6c9f6e9c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry. I0913 23:13:48.417775 14732 consensus_queue.cc:984] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0913 23:13:48.423213 14727 ts_tablet_manager.cc:1113] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090: Time spent starting tablet: real 0.034s user 0.005s sys 0.000s I0913 23:13:48.428645 14733 raft_consensus.cc:472] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0913 23:13:48.429067 14733 raft_consensus.cc:494] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } I0913 23:13:48.432559 14733 leader_election.cc:296] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3:37091), 93a20b201a904b628926e173bbf23463 (127.13.53.1:44785) I0913 23:13:48.434911 14681 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bf89c16ce260470a863711ddbbdb3d68" candidate_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" is_pre_election: true I0913 23:13:48.436908 14681 raft_consensus.cc:2280] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0cad5c4a245f4b3dbd66bb7259f45090 in current term 1: Already voted for candidate 93a20b201a904b628926e173bbf23463 in this term. I0913 23:13:48.434854 14539 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bf89c16ce260470a863711ddbbdb3d68" candidate_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93a20b201a904b628926e173bbf23463" is_pre_election: true I0913 23:13:48.442633 14539 raft_consensus.cc:2311] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0cad5c4a245f4b3dbd66bb7259f45090 for term 1 because replica is either leader or believes a valid leader to be alive. I0913 23:13:48.447594 14571 leader_election.cc:310] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cad5c4a245f4b3dbd66bb7259f45090; no voters: 93a20b201a904b628926e173bbf23463, a66db810508f4e35bac62f5c6c9f6e9c I0913 23:13:48.448524 14733 raft_consensus.cc:2890] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 0 FOLLOWER]: Advancing to term 1 I0913 23:13:48.457686 14733 raft_consensus.cc:2592] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0913 23:13:48.453122 14743 raft_consensus.cc:472] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0913 23:13:48.481982 14734 consensus_queue.cc:984] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0913 23:13:48.478405 14743 raft_consensus.cc:494] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:48.506244 14539 raft_consensus.cc:2890] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [term 0 FOLLOWER]: Advancing to term 1 I0913 23:13:48.516449 14728 tablet_bootstrap.cc:439] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c: No bootstrap required, opened a new log I0913 23:13:48.528724 14743 leader_election.cc:296] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231), a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3:37091) I0913 23:13:48.526006 14728 ts_tablet_manager.cc:1085] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c: Time spent bootstrapping tablet: real 0.196s user 0.020s sys 0.014s I0913 23:13:48.522459 14384 catalog_manager.cc:4079] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 reported cstate change: term changed from 0 to 1, leader changed from to 93a20b201a904b628926e173bbf23463 (127.13.53.1). New cstate: current_term: 1 leader_uuid: "93a20b201a904b628926e173bbf23463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } health_report { overall_health: UNKNOWN } } } I0913 23:13:48.538672 14734 consensus_queue.cc:984] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0913 23:13:48.560665 14680 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "82f65eba7703468ca650e7ca50d39549" candidate_uuid: "93a20b201a904b628926e173bbf23463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" is_pre_election: true I0913 23:13:48.561396 14680 raft_consensus.cc:2311] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 93a20b201a904b628926e173bbf23463 for term 1 because replica is either leader or believes a valid leader to be alive. I0913 23:13:48.567950 14743 consensus_queue.cc:984] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0913 23:13:48.576603 14728 raft_consensus.cc:340] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:48.577484 14728 raft_consensus.cc:366] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0913 23:13:48.577728 14728 raft_consensus.cc:705] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a66db810508f4e35bac62f5c6c9f6e9c, State: Initialized, Role: FOLLOWER I0913 23:13:48.578444 14728 consensus_queue.cc:229] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:48.580178 14728 ts_tablet_manager.cc:1113] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c: Time spent starting tablet: real 0.050s user 0.005s sys 0.000s I0913 23:13:48.597069 14734 raft_consensus.cc:472] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0913 23:13:48.633903 14733 raft_consensus.cc:472] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93a20b201a904b628926e173bbf23463) I0913 23:13:48.606171 14734 raft_consensus.cc:494] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:48.686290 14734 leader_election.cc:296] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93a20b201a904b628926e173bbf23463 (127.13.53.1:44785), 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231) I0913 23:13:48.658651 14733 raft_consensus.cc:494] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:48.695456 14609 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8ad9594f28c64b8ba3dacd888b84312b" candidate_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" is_pre_election: true I0913 23:13:48.721078 14681 raft_consensus.cc:2890] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 0 FOLLOWER]: Advancing to term 1 I0913 23:13:48.721577 14733 leader_election.cc:296] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 93a20b201a904b628926e173bbf23463 (127.13.53.1:44785), a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3:37091) I0913 23:13:48.723754 14733 raft_consensus.cc:472] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0913 23:13:48.726743 14733 raft_consensus.cc:494] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } I0913 23:13:48.732403 14538 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8ad9594f28c64b8ba3dacd888b84312b" candidate_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93a20b201a904b628926e173bbf23463" is_pre_election: true I0913 23:13:48.733266 14538 raft_consensus.cc:2311] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate a66db810508f4e35bac62f5c6c9f6e9c for term 1 because replica is either leader or believes a valid leader to be alive. I0913 23:13:48.734535 14607 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "82f65eba7703468ca650e7ca50d39549" candidate_uuid: "93a20b201a904b628926e173bbf23463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" is_pre_election: true I0913 23:13:48.744997 14762 consensus_queue.cc:984] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s I0913 23:13:48.757369 14679 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8ad9594f28c64b8ba3dacd888b84312b" candidate_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" is_pre_election: true I0913 23:13:48.757046 14762 raft_consensus.cc:472] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a66db810508f4e35bac62f5c6c9f6e9c) I0913 23:13:48.762863 14733 leader_election.cc:296] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3:37091), 93a20b201a904b628926e173bbf23463 (127.13.53.1:44785) I0913 23:13:48.770936 14607 raft_consensus.cc:2325] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93a20b201a904b628926e173bbf23463 for term 1 because replica is already servicing an update from a current leader or another vote. I0913 23:13:48.770335 14680 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bf89c16ce260470a863711ddbbdb3d68" candidate_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" is_pre_election: true I0913 23:13:48.775184 14538 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8ad9594f28c64b8ba3dacd888b84312b" candidate_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93a20b201a904b628926e173bbf23463" is_pre_election: true W0913 23:13:48.783327 14497 leader_election.cc:341] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231): Timed out: RequestConsensusVote RPC to 127.13.53.2:41231 timed out after 0.200s (SENT) I0913 23:13:48.784462 14609 raft_consensus.cc:2325] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a66db810508f4e35bac62f5c6c9f6e9c for term 1 because replica is already servicing an update from a current leader or another vote. I0913 23:13:48.784508 14537 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bf89c16ce260470a863711ddbbdb3d68" candidate_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93a20b201a904b628926e173bbf23463" is_pre_election: true I0913 23:13:48.785459 14537 raft_consensus.cc:2311] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0cad5c4a245f4b3dbd66bb7259f45090 for term 2 because replica is either leader or believes a valid leader to be alive. I0913 23:13:48.785018 14538 raft_consensus.cc:2311] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0cad5c4a245f4b3dbd66bb7259f45090 for term 2 because replica is either leader or believes a valid leader to be alive. I0913 23:13:48.786406 14680 raft_consensus.cc:2325] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0cad5c4a245f4b3dbd66bb7259f45090 for term 2 because replica is already servicing an update from a current leader or another vote. I0913 23:13:48.786331 14643 leader_election.cc:310] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a66db810508f4e35bac62f5c6c9f6e9c; no voters: 0cad5c4a245f4b3dbd66bb7259f45090, 93a20b201a904b628926e173bbf23463 I0913 23:13:48.790127 14497 leader_election.cc:310] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93a20b201a904b628926e173bbf23463; no voters: 0cad5c4a245f4b3dbd66bb7259f45090, a66db810508f4e35bac62f5c6c9f6e9c I0913 23:13:48.790866 14574 leader_election.cc:310] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cad5c4a245f4b3dbd66bb7259f45090; no voters: 93a20b201a904b628926e173bbf23463, a66db810508f4e35bac62f5c6c9f6e9c I0913 23:13:48.767189 14762 raft_consensus.cc:494] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:48.792701 14762 leader_election.cc:296] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231), a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3:37091) I0913 23:13:48.811378 14762 raft_consensus.cc:2592] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0913 23:13:48.814991 14680 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "82f65eba7703468ca650e7ca50d39549" candidate_uuid: "93a20b201a904b628926e173bbf23463" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" is_pre_election: true I0913 23:13:48.815832 14680 raft_consensus.cc:2311] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 93a20b201a904b628926e173bbf23463 for term 2 because replica is either leader or believes a valid leader to be alive. I0913 23:13:48.820348 14609 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "82f65eba7703468ca650e7ca50d39549" candidate_uuid: "93a20b201a904b628926e173bbf23463" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" is_pre_election: true I0913 23:13:48.823071 14772 raft_consensus.cc:472] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a66db810508f4e35bac62f5c6c9f6e9c) I0913 23:13:48.824169 14772 raft_consensus.cc:494] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:48.830135 14609 raft_consensus.cc:2325] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93a20b201a904b628926e173bbf23463 for term 2 because replica is already servicing an update from a current leader or another vote. I0913 23:13:48.830041 14680 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "82f65eba7703468ca650e7ca50d39549" candidate_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" is_pre_election: true I0913 23:13:48.830976 14680 raft_consensus.cc:2311] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0cad5c4a245f4b3dbd66bb7259f45090 for term 2 because replica is either leader or believes a valid leader to be alive. I0913 23:13:48.828009 14772 leader_election.cc:296] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 93a20b201a904b628926e173bbf23463 (127.13.53.1:44785), a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3:37091) I0913 23:13:48.833235 14539 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "82f65eba7703468ca650e7ca50d39549" candidate_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "93a20b201a904b628926e173bbf23463" is_pre_election: true I0913 23:13:48.833644 14497 leader_election.cc:310] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93a20b201a904b628926e173bbf23463; no voters: 0cad5c4a245f4b3dbd66bb7259f45090, a66db810508f4e35bac62f5c6c9f6e9c I0913 23:13:48.834048 14539 raft_consensus.cc:2311] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0cad5c4a245f4b3dbd66bb7259f45090 for term 2 because replica is either leader or believes a valid leader to be alive. I0913 23:13:48.837100 14762 raft_consensus.cc:2592] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0913 23:13:48.837790 14574 leader_election.cc:310] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cad5c4a245f4b3dbd66bb7259f45090; no voters: 93a20b201a904b628926e173bbf23463, a66db810508f4e35bac62f5c6c9f6e9c I0913 23:13:48.838930 14772 raft_consensus.cc:2592] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0913 23:13:48.845201 14769 raft_consensus.cc:472] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b03280a8df7e48f9939e076d6d0138ca) I0913 23:13:48.845822 14769 raft_consensus.cc:494] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:48.847398 14733 raft_consensus.cc:2592] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0913 23:13:48.848278 14768 raft_consensus.cc:2592] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0913 23:13:48.853456 14679 raft_consensus.cc:2325] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0cad5c4a245f4b3dbd66bb7259f45090 for term 2 because replica is already servicing an update from a current leader or another vote. I0913 23:13:48.856822 14571 leader_election.cc:310] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cad5c4a245f4b3dbd66bb7259f45090; no voters: 93a20b201a904b628926e173bbf23463, a66db810508f4e35bac62f5c6c9f6e9c I0913 23:13:48.858764 14733 raft_consensus.cc:2592] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0913 23:13:48.877077 14769 leader_election.cc:296] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46574f27d20e427381df2ea44498c90b (127.13.53.62:40439), b03280a8df7e48f9939e076d6d0138ca (127.13.53.61:35977) I0913 23:13:48.877076 14337 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fab3d6e5bcad48acbb0fe6b928638607" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "46574f27d20e427381df2ea44498c90b" is_pre_election: true I0913 23:13:48.878000 14337 raft_consensus.cc:2311] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fab3d6e5bcad48acbb0fe6b928638607 for term 2 because replica is either leader or believes a valid leader to be alive. I0913 23:13:48.877996 14394 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fab3d6e5bcad48acbb0fe6b928638607" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "b03280a8df7e48f9939e076d6d0138ca" is_pre_election: true I0913 23:13:48.879040 14394 raft_consensus.cc:2311] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate fab3d6e5bcad48acbb0fe6b928638607 for term 2 because replica is either leader or believes a valid leader to be alive. I0913 23:13:48.883358 14424 leader_election.cc:310] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fab3d6e5bcad48acbb0fe6b928638607; no voters: 46574f27d20e427381df2ea44498c90b, b03280a8df7e48f9939e076d6d0138ca I0913 23:13:48.884048 14769 raft_consensus.cc:2592] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0913 23:13:49.217483 14769 raft_consensus.cc:472] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b03280a8df7e48f9939e076d6d0138ca) I0913 23:13:49.217960 14769 raft_consensus.cc:494] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:49.228330 14769 leader_election.cc:296] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46574f27d20e427381df2ea44498c90b (127.13.53.62:40439), b03280a8df7e48f9939e076d6d0138ca (127.13.53.61:35977) I0913 23:13:49.230389 14394 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fab3d6e5bcad48acbb0fe6b928638607" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "b03280a8df7e48f9939e076d6d0138ca" is_pre_election: true I0913 23:13:49.231233 14394 raft_consensus.cc:2311] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate fab3d6e5bcad48acbb0fe6b928638607 for term 2 because replica is either leader or believes a valid leader to be alive. I0913 23:13:49.232652 14337 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fab3d6e5bcad48acbb0fe6b928638607" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "46574f27d20e427381df2ea44498c90b" is_pre_election: true I0913 23:13:49.234596 14337 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fab3d6e5bcad48acbb0fe6b928638607 in term 1. I0913 23:13:49.236948 14423 leader_election.cc:310] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46574f27d20e427381df2ea44498c90b, fab3d6e5bcad48acbb0fe6b928638607; no voters: b03280a8df7e48f9939e076d6d0138ca I0913 23:13:49.240249 14769 raft_consensus.cc:2644] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0913 23:13:49.240558 14769 raft_consensus.cc:472] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b03280a8df7e48f9939e076d6d0138ca) I0913 23:13:49.240833 14769 raft_consensus.cc:2890] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 1 FOLLOWER]: Advancing to term 2 I0913 23:13:49.248397 14769 raft_consensus.cc:494] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:49.252900 14337 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fab3d6e5bcad48acbb0fe6b928638607" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "46574f27d20e427381df2ea44498c90b" I0913 23:13:49.253747 14337 raft_consensus.cc:2311] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fab3d6e5bcad48acbb0fe6b928638607 for term 2 because replica is either leader or believes a valid leader to be alive. I0913 23:13:49.269632 14769 leader_election.cc:296] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [CANDIDATE]: Term 2 election: Requested vote from peers 46574f27d20e427381df2ea44498c90b (127.13.53.62:40439), b03280a8df7e48f9939e076d6d0138ca (127.13.53.61:35977) I0913 23:13:49.273721 14394 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fab3d6e5bcad48acbb0fe6b928638607" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "b03280a8df7e48f9939e076d6d0138ca" I0913 23:13:49.279321 14394 raft_consensus.cc:2311] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 1 LEADER]: Leader election vote request: Denying vote to candidate fab3d6e5bcad48acbb0fe6b928638607 for term 2 because replica is either leader or believes a valid leader to be alive. I0913 23:13:49.280237 14424 leader_election.cc:310] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fab3d6e5bcad48acbb0fe6b928638607; no voters: 46574f27d20e427381df2ea44498c90b, b03280a8df7e48f9939e076d6d0138ca I0913 23:13:49.281949 14769 raft_consensus.cc:2592] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority I0913 23:13:49.324386 14451 raft_consensus.cc:1149] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 2 FOLLOWER]: Rejecting Update request from peer b03280a8df7e48f9939e076d6d0138ca for earlier term 1. Current term is 2. Ops: [] I0913 23:13:49.326227 14786 consensus_queue.cc:995] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 }, Status: INVALID_TERM, Last received: 1.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I0913 23:13:49.327464 14786 raft_consensus.cc:2885] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 1 LEADER]: Stepping down as leader of term 1 I0913 23:13:49.327831 14786 raft_consensus.cc:705] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b03280a8df7e48f9939e076d6d0138ca, State: Running, Role: LEADER I0913 23:13:49.328478 14786 consensus_queue.cc:229] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:49.329581 14786 raft_consensus.cc:2890] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 1 FOLLOWER]: Advancing to term 2 I0913 23:13:49.482646 14794 raft_consensus.cc:472] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b03280a8df7e48f9939e076d6d0138ca) I0913 23:13:49.483050 14794 raft_consensus.cc:494] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:49.484388 14794 leader_election.cc:296] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b03280a8df7e48f9939e076d6d0138ca (127.13.53.61:35977), fab3d6e5bcad48acbb0fe6b928638607 (127.13.53.60:42667) I0913 23:13:49.485211 14394 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46574f27d20e427381df2ea44498c90b" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "b03280a8df7e48f9939e076d6d0138ca" is_pre_election: true I0913 23:13:49.485334 14451 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46574f27d20e427381df2ea44498c90b" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "fab3d6e5bcad48acbb0fe6b928638607" is_pre_election: true I0913 23:13:49.485868 14394 raft_consensus.cc:2350] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46574f27d20e427381df2ea44498c90b in term 2. I0913 23:13:49.486099 14451 raft_consensus.cc:2280] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46574f27d20e427381df2ea44498c90b in current term 2: Already voted for candidate fab3d6e5bcad48acbb0fe6b928638607 in this term. I0913 23:13:49.486838 14310 leader_election.cc:310] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46574f27d20e427381df2ea44498c90b, b03280a8df7e48f9939e076d6d0138ca; no voters: I0913 23:13:49.487550 14794 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 1 FOLLOWER]: Leader pre-election won for term 2 I0913 23:13:49.487836 14794 raft_consensus.cc:472] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 1 FOLLOWER]: Starting leader election (detected failure of leader b03280a8df7e48f9939e076d6d0138ca) I0913 23:13:49.488095 14794 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 1 FOLLOWER]: Advancing to term 2 I0913 23:13:49.493499 14794 raft_consensus.cc:494] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:49.495606 14794 leader_election.cc:296] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [CANDIDATE]: Term 2 election: Requested vote from peers b03280a8df7e48f9939e076d6d0138ca (127.13.53.61:35977), fab3d6e5bcad48acbb0fe6b928638607 (127.13.53.60:42667) I0913 23:13:49.495853 14394 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46574f27d20e427381df2ea44498c90b" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "b03280a8df7e48f9939e076d6d0138ca" I0913 23:13:49.496762 14451 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46574f27d20e427381df2ea44498c90b" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "fab3d6e5bcad48acbb0fe6b928638607" I0913 23:13:49.497468 14451 raft_consensus.cc:2280] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 46574f27d20e427381df2ea44498c90b in current term 2: Already voted for candidate fab3d6e5bcad48acbb0fe6b928638607 in this term. I0913 23:13:49.500764 14394 raft_consensus.cc:2350] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46574f27d20e427381df2ea44498c90b in term 2. I0913 23:13:49.501703 14310 leader_election.cc:310] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46574f27d20e427381df2ea44498c90b, b03280a8df7e48f9939e076d6d0138ca; no voters: fab3d6e5bcad48acbb0fe6b928638607 I0913 23:13:49.502442 14794 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 2 FOLLOWER]: Leader election won for term 2 I0913 23:13:49.510782 14794 raft_consensus.cc:667] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 2 LEADER]: Becoming Leader. State: Replica: 46574f27d20e427381df2ea44498c90b, State: Running, Role: LEADER I0913 23:13:49.511513 14794 consensus_queue.cc:206] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:49.513723 14795 sys_catalog.cc:340] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46574f27d20e427381df2ea44498c90b. Latest consensus state: current_term: 2 leader_uuid: "46574f27d20e427381df2ea44498c90b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:49.514369 14795 sys_catalog.cc:343] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: This master's current role is: LEADER I0913 23:13:49.515609 14796 catalog_manager.cc:1071] Loading table and tablet metadata into memory... I0913 23:13:49.518471 14796 catalog_manager.cc:372] Loaded metadata for table test-table [id=334306099b944525a47cb78bb2cafd10] I0913 23:13:49.521150 14796 catalog_manager.cc:426] Loaded metadata for tablet 82f65eba7703468ca650e7ca50d39549 (table test-table [id=334306099b944525a47cb78bb2cafd10]) I0913 23:13:49.522006 14796 catalog_manager.cc:426] Loaded metadata for tablet 8ad9594f28c64b8ba3dacd888b84312b (table test-table [id=334306099b944525a47cb78bb2cafd10]) I0913 23:13:49.522822 14796 catalog_manager.cc:426] Loaded metadata for tablet bf89c16ce260470a863711ddbbdb3d68 (table test-table [id=334306099b944525a47cb78bb2cafd10]) I0913 23:13:49.523319 14796 catalog_manager.cc:1082] Initializing Kudu internal certificate authority... I0913 23:13:49.526104 14796 catalog_manager.cc:1091] Loading token signing keys... I0913 23:13:49.528713 14796 catalog_manager.cc:4405] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b: Loaded TSK: 0 I0913 23:13:49.559767 14769 raft_consensus.cc:472] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0913 23:13:49.560227 14769 raft_consensus.cc:494] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:49.563153 14769 leader_election.cc:296] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46574f27d20e427381df2ea44498c90b (127.13.53.62:40439), b03280a8df7e48f9939e076d6d0138ca (127.13.53.61:35977) I0913 23:13:49.564575 14337 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fab3d6e5bcad48acbb0fe6b928638607" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "46574f27d20e427381df2ea44498c90b" is_pre_election: true I0913 23:13:49.565315 14337 raft_consensus.cc:2311] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate fab3d6e5bcad48acbb0fe6b928638607 for term 3 because replica is either leader or believes a valid leader to be alive. I0913 23:13:49.565088 14394 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fab3d6e5bcad48acbb0fe6b928638607" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "b03280a8df7e48f9939e076d6d0138ca" is_pre_election: true I0913 23:13:49.565716 14394 raft_consensus.cc:2350] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fab3d6e5bcad48acbb0fe6b928638607 in term 2. I0913 23:13:49.566721 14424 leader_election.cc:310] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b03280a8df7e48f9939e076d6d0138ca, fab3d6e5bcad48acbb0fe6b928638607; no voters: 46574f27d20e427381df2ea44498c90b I0913 23:13:49.567404 14769 raft_consensus.cc:2644] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 2 FOLLOWER]: Leader pre-election won for term 3 I0913 23:13:49.567147 14539 tablet_service.cc:1440] Received LeaderStepDown RPC: tablet_id: "bf89c16ce260470a863711ddbbdb3d68" dest_uuid: "93a20b201a904b628926e173bbf23463" from {username='slave'} at 127.0.0.1:60436 I0913 23:13:49.567651 14769 raft_consensus.cc:472] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0913 23:13:49.567862 14539 raft_consensus.cc:569] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [term 1 LEADER]: Received request to step down I0913 23:13:49.567924 14769 raft_consensus.cc:2890] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 2 FOLLOWER]: Advancing to term 3 I0913 23:13:49.568138 14539 raft_consensus.cc:2885] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [term 1 LEADER]: Stepping down as leader of term 1 I0913 23:13:49.568372 14539 raft_consensus.cc:705] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 93a20b201a904b628926e173bbf23463, State: Running, Role: LEADER I0913 23:13:49.568866 14539 consensus_queue.cc:229] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } I0913 23:13:49.569701 14539 raft_consensus.cc:2890] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [term 1 FOLLOWER]: Advancing to term 2 I0913 23:13:49.572069 14769 raft_consensus.cc:494] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:49.573910 14769 leader_election.cc:296] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [CANDIDATE]: Term 3 election: Requested vote from peers 46574f27d20e427381df2ea44498c90b (127.13.53.62:40439), b03280a8df7e48f9939e076d6d0138ca (127.13.53.61:35977) I0913 23:13:49.573982 14337 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fab3d6e5bcad48acbb0fe6b928638607" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "46574f27d20e427381df2ea44498c90b" I0913 23:13:49.574038 14394 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fab3d6e5bcad48acbb0fe6b928638607" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "b03280a8df7e48f9939e076d6d0138ca" I0913 23:13:49.574563 14337 raft_consensus.cc:2311] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 2 LEADER]: Leader election vote request: Denying vote to candidate fab3d6e5bcad48acbb0fe6b928638607 for term 3 because replica is either leader or believes a valid leader to be alive. I0913 23:13:49.574599 14394 raft_consensus.cc:2890] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 2 FOLLOWER]: Advancing to term 3 I0913 23:13:49.577913 14394 raft_consensus.cc:2350] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fab3d6e5bcad48acbb0fe6b928638607 in term 3. I0913 23:13:49.578740 14424 leader_election.cc:310] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b03280a8df7e48f9939e076d6d0138ca, fab3d6e5bcad48acbb0fe6b928638607; no voters: 46574f27d20e427381df2ea44498c90b I0913 23:13:49.579360 14769 raft_consensus.cc:2644] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 3 FOLLOWER]: Leader election won for term 3 I0913 23:13:49.580410 14769 raft_consensus.cc:667] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 3 LEADER]: Becoming Leader. State: Replica: fab3d6e5bcad48acbb0fe6b928638607, State: Running, Role: LEADER I0913 23:13:49.580830 14769 consensus_queue.cc:206] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:49.582610 14800 sys_catalog.cc:340] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fab3d6e5bcad48acbb0fe6b928638607. Latest consensus state: current_term: 3 leader_uuid: "fab3d6e5bcad48acbb0fe6b928638607" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:49.583135 14800 sys_catalog.cc:343] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: This master's current role is: LEADER I0913 23:13:49.584697 14801 catalog_manager.cc:1071] Loading table and tablet metadata into memory... I0913 23:13:49.587283 14801 catalog_manager.cc:372] Loaded metadata for table test-table [id=334306099b944525a47cb78bb2cafd10] I0913 23:13:49.589622 14801 catalog_manager.cc:426] Loaded metadata for tablet 82f65eba7703468ca650e7ca50d39549 (table test-table [id=334306099b944525a47cb78bb2cafd10]) I0913 23:13:49.590252 14801 catalog_manager.cc:426] Loaded metadata for tablet 8ad9594f28c64b8ba3dacd888b84312b (table test-table [id=334306099b944525a47cb78bb2cafd10]) I0913 23:13:49.590807 14801 catalog_manager.cc:426] Loaded metadata for tablet bf89c16ce260470a863711ddbbdb3d68 (table test-table [id=334306099b944525a47cb78bb2cafd10]) I0913 23:13:49.591342 14801 catalog_manager.cc:1082] Initializing Kudu internal certificate authority... I0913 23:13:49.592550 14451 raft_consensus.cc:1149] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 3 LEADER]: Rejecting Update request from peer 46574f27d20e427381df2ea44498c90b for earlier term 2. Current term is 3. Ops: [] I0913 23:13:49.593834 14795 consensus_queue.cc:995] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I0913 23:13:49.594451 14795 consensus_queue.cc:1162] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [LEADER]: Peer fab3d6e5bcad48acbb0fe6b928638607 log is divergent from this leader: its last log entry 3.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8 I0913 23:13:49.594525 14801 catalog_manager.cc:1091] Loading token signing keys... I0913 23:13:49.594919 14794 raft_consensus.cc:2885] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 2 LEADER]: Stepping down as leader of term 2 I0913 23:13:49.595147 14794 raft_consensus.cc:705] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 46574f27d20e427381df2ea44498c90b, State: Running, Role: LEADER I0913 23:13:49.595635 14794 consensus_queue.cc:229] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.9, Last appended by leader: 9, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:49.596614 14794 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 2 FOLLOWER]: Advancing to term 3 I0913 23:13:49.596909 14801 catalog_manager.cc:4405] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607: Loaded TSK: 0 I0913 23:13:49.636487 14636 heartbeater.cc:447] Master 127.13.53.62:40439 was elected leader, sending a full tablet report... I0913 23:13:49.667798 14337 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 3 FOLLOWER]: Refusing update from remote peer fab3d6e5bcad48acbb0fe6b928638607: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 9. (term mismatch) I0913 23:13:49.668063 14337 pending_rounds.cc:78] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b: Aborting all transactions after (but not including) 8 I0913 23:13:49.668215 14337 pending_rounds.cc:99] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b: Aborting uncommitted NO_OP operation due to leader change: 2.9 I0913 23:13:49.668429 14337 raft_consensus.cc:2727] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader I0913 23:13:49.669600 14800 consensus_queue.cc:984] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I0913 23:13:49.671731 14394 raft_consensus.cc:1184] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 3 FOLLOWER]: Refusing update from remote peer fab3d6e5bcad48acbb0fe6b928638607: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch) I0913 23:13:49.672724 14800 consensus_queue.cc:984] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I0913 23:13:49.672791 14794 sys_catalog.cc:340] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fab3d6e5bcad48acbb0fe6b928638607. Latest consensus state: current_term: 3 leader_uuid: "fab3d6e5bcad48acbb0fe6b928638607" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:49.673477 14794 sys_catalog.cc:343] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: This master's current role is: FOLLOWER I0913 23:13:49.676514 14786 sys_catalog.cc:340] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader fab3d6e5bcad48acbb0fe6b928638607. Latest consensus state: current_term: 3 leader_uuid: "fab3d6e5bcad48acbb0fe6b928638607" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:49.677834 14786 sys_catalog.cc:343] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [sys.catalog]: This master's current role is: FOLLOWER I0913 23:13:49.679524 14800 sys_catalog.cc:340] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "fab3d6e5bcad48acbb0fe6b928638607" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:49.680310 14800 sys_catalog.cc:343] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: This master's current role is: LEADER I0913 23:13:49.680652 14794 sys_catalog.cc:340] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "fab3d6e5bcad48acbb0fe6b928638607" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:49.684319 14794 sys_catalog.cc:343] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: This master's current role is: FOLLOWER I0913 23:13:49.685847 14806 sys_catalog.cc:340] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "fab3d6e5bcad48acbb0fe6b928638607" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:49.686844 14806 sys_catalog.cc:343] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: This master's current role is: LEADER I0913 23:13:49.688910 14786 sys_catalog.cc:340] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "fab3d6e5bcad48acbb0fe6b928638607" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:49.689700 14786 sys_catalog.cc:343] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [sys.catalog]: This master's current role is: FOLLOWER I0913 23:13:49.728154 14768 raft_consensus.cc:472] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93a20b201a904b628926e173bbf23463) I0913 23:13:49.728710 14768 raft_consensus.cc:494] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } I0913 23:13:49.729990 14768 leader_election.cc:296] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 93a20b201a904b628926e173bbf23463 (127.13.53.1:44785), 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231) I0913 23:13:49.730667 14539 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bf89c16ce260470a863711ddbbdb3d68" candidate_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "93a20b201a904b628926e173bbf23463" is_pre_election: true I0913 23:13:49.730917 14608 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bf89c16ce260470a863711ddbbdb3d68" candidate_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" is_pre_election: true I0913 23:13:49.731345 14539 raft_consensus.cc:2350] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a66db810508f4e35bac62f5c6c9f6e9c in term 2. I0913 23:13:49.731571 14608 raft_consensus.cc:2311] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a66db810508f4e35bac62f5c6c9f6e9c for term 2 because replica is either leader or believes a valid leader to be alive. I0913 23:13:49.732453 14645 leader_election.cc:310] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93a20b201a904b628926e173bbf23463, a66db810508f4e35bac62f5c6c9f6e9c; no voters: I0913 23:13:49.733144 14768 raft_consensus.cc:2644] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [term 1 FOLLOWER]: Leader pre-election won for term 2 I0913 23:13:49.733420 14768 raft_consensus.cc:472] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 93a20b201a904b628926e173bbf23463) I0913 23:13:49.733736 14768 raft_consensus.cc:2890] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [term 1 FOLLOWER]: Advancing to term 2 I0913 23:13:49.734268 14704 heartbeater.cc:447] Master 127.13.53.60:42667 was elected leader, sending a full tablet report... I0913 23:13:49.738647 14768 raft_consensus.cc:494] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } I0913 23:13:49.739296 14562 heartbeater.cc:447] Master 127.13.53.60:42667 was elected leader, sending a full tablet report... I0913 23:13:49.740805 14539 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bf89c16ce260470a863711ddbbdb3d68" candidate_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "93a20b201a904b628926e173bbf23463" I0913 23:13:49.740960 14608 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bf89c16ce260470a863711ddbbdb3d68" candidate_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" I0913 23:13:49.741657 14608 raft_consensus.cc:2311] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a66db810508f4e35bac62f5c6c9f6e9c for term 2 because replica is either leader or believes a valid leader to be alive. I0913 23:13:49.742231 14633 heartbeater.cc:447] Master 127.13.53.60:42667 was elected leader, sending a full tablet report... I0913 23:13:49.744292 14768 leader_election.cc:296] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [CANDIDATE]: Term 2 election: Requested vote from peers 93a20b201a904b628926e173bbf23463 (127.13.53.1:44785), 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231) I0913 23:13:49.746510 14539 raft_consensus.cc:2350] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a66db810508f4e35bac62f5c6c9f6e9c in term 2. I0913 23:13:49.747431 14645 leader_election.cc:310] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93a20b201a904b628926e173bbf23463, a66db810508f4e35bac62f5c6c9f6e9c; no voters: 0cad5c4a245f4b3dbd66bb7259f45090 I0913 23:13:49.748452 14768 raft_consensus.cc:2644] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [term 2 FOLLOWER]: Leader election won for term 2 I0913 23:13:49.749574 14768 raft_consensus.cc:667] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [term 2 LEADER]: Becoming Leader. State: Replica: a66db810508f4e35bac62f5c6c9f6e9c, State: Running, Role: LEADER I0913 23:13:49.750147 14768 consensus_queue.cc:206] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } I0913 23:13:49.755431 14439 catalog_manager.cc:4079] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c reported cstate change: term changed from 1 to 2, leader changed from 93a20b201a904b628926e173bbf23463 (127.13.53.1) to a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3). New cstate: current_term: 2 leader_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } health_report { overall_health: UNKNOWN } } } I0913 23:13:49.765890 14809 raft_consensus.cc:472] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93a20b201a904b628926e173bbf23463) I0913 23:13:49.766644 14809 raft_consensus.cc:494] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } I0913 23:13:49.770130 14679 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bf89c16ce260470a863711ddbbdb3d68" candidate_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" is_pre_election: true I0913 23:13:49.770844 14679 raft_consensus.cc:2311] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 0cad5c4a245f4b3dbd66bb7259f45090 for term 2 because replica is either leader or believes a valid leader to be alive. I0913 23:13:49.771083 14539 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bf89c16ce260470a863711ddbbdb3d68" candidate_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "93a20b201a904b628926e173bbf23463" is_pre_election: true I0913 23:13:49.771781 14539 raft_consensus.cc:2280] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0cad5c4a245f4b3dbd66bb7259f45090 in current term 2: Already voted for candidate a66db810508f4e35bac62f5c6c9f6e9c in this term. I0913 23:13:49.772753 14574 leader_election.cc:310] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cad5c4a245f4b3dbd66bb7259f45090; no voters: 93a20b201a904b628926e173bbf23463, a66db810508f4e35bac62f5c6c9f6e9c I0913 23:13:49.773773 14809 leader_election.cc:296] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3:37091), 93a20b201a904b628926e173bbf23463 (127.13.53.1:44785) I0913 23:13:49.780095 14809 raft_consensus.cc:2890] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 1 FOLLOWER]: Advancing to term 2 I0913 23:13:49.790948 14809 raft_consensus.cc:2592] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0913 23:13:49.795472 14539 tablet_service.cc:1440] Received LeaderStepDown RPC: tablet_id: "8ad9594f28c64b8ba3dacd888b84312b" dest_uuid: "93a20b201a904b628926e173bbf23463" from {username='slave'} at 127.0.0.1:60436 I0913 23:13:49.796027 14539 raft_consensus.cc:569] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [term 1 LEADER]: Received request to step down I0913 23:13:49.796248 14539 raft_consensus.cc:2885] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [term 1 LEADER]: Stepping down as leader of term 1 I0913 23:13:49.796417 14539 raft_consensus.cc:705] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 93a20b201a904b628926e173bbf23463, State: Running, Role: LEADER I0913 23:13:49.796854 14539 consensus_queue.cc:229] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:49.797654 14539 raft_consensus.cc:2890] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [term 1 FOLLOWER]: Advancing to term 2 I0913 23:13:49.850136 14608 raft_consensus.cc:1184] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 2 FOLLOWER]: Refusing update from remote peer a66db810508f4e35bac62f5c6c9f6e9c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch) I0913 23:13:49.851148 14768 consensus_queue.cc:984] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0913 23:13:49.856140 14539 raft_consensus.cc:1184] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [term 2 FOLLOWER]: Refusing update from remote peer a66db810508f4e35bac62f5c6c9f6e9c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch) I0913 23:13:49.857285 14768 consensus_queue.cc:984] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0913 23:13:49.884825 14480 catalog_manager.cc:1155] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0913 23:13:49.958525 14808 raft_consensus.cc:472] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93a20b201a904b628926e173bbf23463) I0913 23:13:49.959065 14809 raft_consensus.cc:472] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 93a20b201a904b628926e173bbf23463) I0913 23:13:49.959055 14808 raft_consensus.cc:494] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:49.959619 14809 raft_consensus.cc:494] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:49.961036 14808 leader_election.cc:296] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 93a20b201a904b628926e173bbf23463 (127.13.53.1:44785), 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231) I0913 23:13:49.961906 14679 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8ad9594f28c64b8ba3dacd888b84312b" candidate_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" is_pre_election: true I0913 23:13:49.962524 14679 raft_consensus.cc:2350] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cad5c4a245f4b3dbd66bb7259f45090 in term 1. I0913 23:13:49.962487 14538 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8ad9594f28c64b8ba3dacd888b84312b" candidate_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "93a20b201a904b628926e173bbf23463" is_pre_election: true I0913 23:13:49.962889 14809 leader_election.cc:296] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 93a20b201a904b628926e173bbf23463 (127.13.53.1:44785), a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3:37091) I0913 23:13:49.963289 14538 raft_consensus.cc:2350] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a66db810508f4e35bac62f5c6c9f6e9c in term 2. I0913 23:13:49.963594 14571 leader_election.cc:310] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cad5c4a245f4b3dbd66bb7259f45090, a66db810508f4e35bac62f5c6c9f6e9c; no voters: I0913 23:13:49.963788 14608 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8ad9594f28c64b8ba3dacd888b84312b" candidate_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" is_pre_election: true I0913 23:13:49.963891 14539 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8ad9594f28c64b8ba3dacd888b84312b" candidate_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "93a20b201a904b628926e173bbf23463" is_pre_election: true I0913 23:13:49.964375 14608 raft_consensus.cc:2350] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a66db810508f4e35bac62f5c6c9f6e9c in term 1. I0913 23:13:49.964493 14539 raft_consensus.cc:2350] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cad5c4a245f4b3dbd66bb7259f45090 in term 2. I0913 23:13:49.964392 14645 leader_election.cc:310] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93a20b201a904b628926e173bbf23463, a66db810508f4e35bac62f5c6c9f6e9c; no voters: I0913 23:13:49.965214 14809 raft_consensus.cc:2644] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0913 23:13:49.965500 14808 raft_consensus.cc:2644] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 1 FOLLOWER]: Leader pre-election won for term 2 I0913 23:13:49.965493 14809 raft_consensus.cc:472] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 93a20b201a904b628926e173bbf23463) I0913 23:13:49.965925 14808 raft_consensus.cc:472] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 93a20b201a904b628926e173bbf23463) I0913 23:13:49.966092 14809 raft_consensus.cc:2890] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [term 1 FOLLOWER]: Advancing to term 2 I0913 23:13:49.966293 14808 raft_consensus.cc:2890] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 1 FOLLOWER]: Advancing to term 2 I0913 23:13:49.970971 14808 raft_consensus.cc:494] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:49.970968 14809 raft_consensus.cc:494] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:49.972555 14809 leader_election.cc:296] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [CANDIDATE]: Term 2 election: Requested vote from peers 93a20b201a904b628926e173bbf23463 (127.13.53.1:44785), a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3:37091) I0913 23:13:49.973129 14539 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8ad9594f28c64b8ba3dacd888b84312b" candidate_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "93a20b201a904b628926e173bbf23463" I0913 23:13:49.973325 14679 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8ad9594f28c64b8ba3dacd888b84312b" candidate_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" I0913 23:13:49.974099 14679 raft_consensus.cc:2280] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0cad5c4a245f4b3dbd66bb7259f45090 in current term 2: Already voted for candidate a66db810508f4e35bac62f5c6c9f6e9c in this term. I0913 23:13:49.975782 14538 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8ad9594f28c64b8ba3dacd888b84312b" candidate_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "93a20b201a904b628926e173bbf23463" I0913 23:13:49.977424 14808 leader_election.cc:296] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [CANDIDATE]: Term 2 election: Requested vote from peers 93a20b201a904b628926e173bbf23463 (127.13.53.1:44785), 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231) I0913 23:13:49.977334 14608 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8ad9594f28c64b8ba3dacd888b84312b" candidate_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" I0913 23:13:49.978118 14608 raft_consensus.cc:2280] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a66db810508f4e35bac62f5c6c9f6e9c in current term 2: Already voted for candidate 0cad5c4a245f4b3dbd66bb7259f45090 in this term. I0913 23:13:49.982694 14539 raft_consensus.cc:2350] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a66db810508f4e35bac62f5c6c9f6e9c in term 2. I0913 23:13:49.983223 14538 raft_consensus.cc:2325] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0cad5c4a245f4b3dbd66bb7259f45090 for term 2 because replica is already servicing an update from a current leader or another vote. I0913 23:13:49.983613 14645 leader_election.cc:310] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93a20b201a904b628926e173bbf23463, a66db810508f4e35bac62f5c6c9f6e9c; no voters: 0cad5c4a245f4b3dbd66bb7259f45090 I0913 23:13:49.984279 14808 raft_consensus.cc:2644] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 2 FOLLOWER]: Leader election won for term 2 I0913 23:13:49.984442 14574 leader_election.cc:310] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cad5c4a245f4b3dbd66bb7259f45090; no voters: 93a20b201a904b628926e173bbf23463, a66db810508f4e35bac62f5c6c9f6e9c I0913 23:13:49.984717 14808 raft_consensus.cc:667] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 2 LEADER]: Becoming Leader. State: Replica: a66db810508f4e35bac62f5c6c9f6e9c, State: Running, Role: LEADER I0913 23:13:49.985493 14809 raft_consensus.cc:2592] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority I0913 23:13:49.985255 14808 consensus_queue.cc:206] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:49.989892 14439 catalog_manager.cc:4079] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c reported cstate change: term changed from 1 to 2, leader changed from 93a20b201a904b628926e173bbf23463 (127.13.53.1) to a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3). New cstate: current_term: 2 leader_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } health_report { overall_health: HEALTHY } } } I0913 23:13:50.023419 14679 tablet_service.cc:1440] Received LeaderStepDown RPC: tablet_id: "8ad9594f28c64b8ba3dacd888b84312b" dest_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" from {username='slave'} at 127.0.0.1:59276 I0913 23:13:50.023901 14679 raft_consensus.cc:569] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 2 LEADER]: Received request to step down I0913 23:13:50.024101 14679 raft_consensus.cc:2885] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 2 LEADER]: Stepping down as leader of term 2 I0913 23:13:50.024271 14679 raft_consensus.cc:705] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a66db810508f4e35bac62f5c6c9f6e9c, State: Running, Role: LEADER I0913 23:13:50.024724 14679 consensus_queue.cc:229] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:50.025424 14679 raft_consensus.cc:2890] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 2 FOLLOWER]: Advancing to term 3 I0913 23:13:50.214269 14762 raft_consensus.cc:472] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0913 23:13:50.214746 14762 raft_consensus.cc:494] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:50.216140 14762 leader_election.cc:296] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231), a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3:37091) I0913 23:13:50.216753 14608 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8ad9594f28c64b8ba3dacd888b84312b" candidate_uuid: "93a20b201a904b628926e173bbf23463" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" is_pre_election: true I0913 23:13:50.217010 14679 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8ad9594f28c64b8ba3dacd888b84312b" candidate_uuid: "93a20b201a904b628926e173bbf23463" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" is_pre_election: true I0913 23:13:50.217372 14608 raft_consensus.cc:2350] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93a20b201a904b628926e173bbf23463 in term 2. I0913 23:13:50.217787 14679 raft_consensus.cc:2297] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93a20b201a904b628926e173bbf23463 for term 3 because replica has last-logged OpId of term: 2 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2. I0913 23:13:50.218286 14497 leader_election.cc:310] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cad5c4a245f4b3dbd66bb7259f45090, 93a20b201a904b628926e173bbf23463; no voters: I0913 23:13:50.219099 14762 raft_consensus.cc:2644] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [term 2 FOLLOWER]: Leader pre-election won for term 3 I0913 23:13:50.219370 14762 raft_consensus.cc:472] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0913 23:13:50.221230 14762 raft_consensus.cc:2890] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [term 2 FOLLOWER]: Advancing to term 3 I0913 23:13:50.225587 14762 raft_consensus.cc:494] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:50.226753 14762 leader_election.cc:296] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [CANDIDATE]: Term 3 election: Requested vote from peers 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231), a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3:37091) I0913 23:13:50.228215 14608 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8ad9594f28c64b8ba3dacd888b84312b" candidate_uuid: "93a20b201a904b628926e173bbf23463" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" I0913 23:13:50.228875 14608 raft_consensus.cc:2890] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [term 2 FOLLOWER]: Advancing to term 3 I0913 23:13:50.229041 14679 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8ad9594f28c64b8ba3dacd888b84312b" candidate_uuid: "93a20b201a904b628926e173bbf23463" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" I0913 23:13:50.229810 14679 raft_consensus.cc:2297] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 93a20b201a904b628926e173bbf23463 for term 3 because replica has last-logged OpId of term: 2 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2. I0913 23:13:50.230885 14679 tablet_service.cc:1440] Received LeaderStepDown RPC: tablet_id: "bf89c16ce260470a863711ddbbdb3d68" dest_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" from {username='slave'} at 127.0.0.1:59276 I0913 23:13:50.231281 14679 raft_consensus.cc:569] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [term 2 LEADER]: Received request to step down I0913 23:13:50.231504 14679 raft_consensus.cc:2885] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [term 2 LEADER]: Stepping down as leader of term 2 I0913 23:13:50.231700 14679 raft_consensus.cc:705] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a66db810508f4e35bac62f5c6c9f6e9c, State: Running, Role: LEADER I0913 23:13:50.232216 14679 consensus_queue.cc:229] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } I0913 23:13:50.233271 14679 raft_consensus.cc:2890] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [term 2 FOLLOWER]: Advancing to term 3 I0913 23:13:50.239029 14608 raft_consensus.cc:2350] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93a20b201a904b628926e173bbf23463 in term 3. I0913 23:13:50.240358 14497 leader_election.cc:310] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cad5c4a245f4b3dbd66bb7259f45090, 93a20b201a904b628926e173bbf23463; no voters: a66db810508f4e35bac62f5c6c9f6e9c I0913 23:13:50.241379 14762 raft_consensus.cc:2644] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [term 3 FOLLOWER]: Leader election won for term 3 I0913 23:13:50.241993 14762 raft_consensus.cc:667] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [term 3 LEADER]: Becoming Leader. State: Replica: 93a20b201a904b628926e173bbf23463, State: Running, Role: LEADER I0913 23:13:50.242492 14762 consensus_queue.cc:206] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:50.247400 14439 catalog_manager.cc:4079] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 reported cstate change: term changed from 2 to 3, leader changed from a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3) to 93a20b201a904b628926e173bbf23463 (127.13.53.1). New cstate: current_term: 3 leader_uuid: "93a20b201a904b628926e173bbf23463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } health_report { overall_health: UNKNOWN } } } I0913 23:13:50.339326 14608 raft_consensus.cc:1184] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [term 3 FOLLOWER]: Refusing update from remote peer 93a20b201a904b628926e173bbf23463: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch) I0913 23:13:50.340301 14762 consensus_queue.cc:984] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0913 23:13:50.345691 14679 raft_consensus.cc:1184] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 3 FOLLOWER]: Refusing update from remote peer 93a20b201a904b628926e173bbf23463: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 3. (term mismatch) I0913 23:13:50.346040 14679 pending_rounds.cc:78] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c: Aborting all transactions after (but not including) 2 I0913 23:13:50.346303 14679 pending_rounds.cc:99] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c: Aborting uncommitted NO_OP operation due to leader change: 2.3 I0913 23:13:50.346549 14679 raft_consensus.cc:2727] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader I0913 23:13:50.347689 14762 consensus_queue.cc:984] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0913 23:13:50.388778 14821 raft_consensus.cc:472] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a66db810508f4e35bac62f5c6c9f6e9c) I0913 23:13:50.389186 14821 raft_consensus.cc:494] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } I0913 23:13:50.391018 14821 leader_election.cc:296] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3:37091), 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231) I0913 23:13:50.391382 14679 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bf89c16ce260470a863711ddbbdb3d68" candidate_uuid: "93a20b201a904b628926e173bbf23463" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" is_pre_election: true I0913 23:13:50.391898 14679 raft_consensus.cc:2350] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93a20b201a904b628926e173bbf23463 in term 3. I0913 23:13:50.391836 14608 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bf89c16ce260470a863711ddbbdb3d68" candidate_uuid: "93a20b201a904b628926e173bbf23463" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" is_pre_election: true I0913 23:13:50.392453 14608 raft_consensus.cc:2311] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93a20b201a904b628926e173bbf23463 for term 3 because replica is either leader or believes a valid leader to be alive. I0913 23:13:50.392957 14496 leader_election.cc:310] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93a20b201a904b628926e173bbf23463, a66db810508f4e35bac62f5c6c9f6e9c; no voters: I0913 23:13:50.393743 14821 raft_consensus.cc:2644] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [term 2 FOLLOWER]: Leader pre-election won for term 3 I0913 23:13:50.393970 14821 raft_consensus.cc:472] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a66db810508f4e35bac62f5c6c9f6e9c) I0913 23:13:50.394209 14821 raft_consensus.cc:2890] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [term 2 FOLLOWER]: Advancing to term 3 I0913 23:13:50.398166 14821 raft_consensus.cc:494] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } I0913 23:13:50.399396 14821 leader_election.cc:296] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [CANDIDATE]: Term 3 election: Requested vote from peers a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3:37091), 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231) I0913 23:13:50.400204 14608 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bf89c16ce260470a863711ddbbdb3d68" candidate_uuid: "93a20b201a904b628926e173bbf23463" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" I0913 23:13:50.400202 14679 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bf89c16ce260470a863711ddbbdb3d68" candidate_uuid: "93a20b201a904b628926e173bbf23463" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" I0913 23:13:50.400974 14608 raft_consensus.cc:2311] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 93a20b201a904b628926e173bbf23463 for term 3 because replica is either leader or believes a valid leader to be alive. I0913 23:13:50.404731 14679 raft_consensus.cc:2350] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93a20b201a904b628926e173bbf23463 in term 3. I0913 23:13:50.405741 14496 leader_election.cc:310] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93a20b201a904b628926e173bbf23463, a66db810508f4e35bac62f5c6c9f6e9c; no voters: 0cad5c4a245f4b3dbd66bb7259f45090 I0913 23:13:50.406510 14821 raft_consensus.cc:2644] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [term 3 FOLLOWER]: Leader election won for term 3 I0913 23:13:50.406853 14821 raft_consensus.cc:667] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [term 3 LEADER]: Becoming Leader. State: Replica: 93a20b201a904b628926e173bbf23463, State: Running, Role: LEADER I0913 23:13:50.407407 14821 consensus_queue.cc:206] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } I0913 23:13:50.412864 14441 catalog_manager.cc:4079] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 reported cstate change: term changed from 2 to 3, leader changed from a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3) to 93a20b201a904b628926e173bbf23463 (127.13.53.1). New cstate: current_term: 3 leader_uuid: "93a20b201a904b628926e173bbf23463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } health_report { overall_health: UNKNOWN } } } I0913 23:13:50.423377 14809 raft_consensus.cc:472] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a66db810508f4e35bac62f5c6c9f6e9c) I0913 23:13:50.423856 14809 raft_consensus.cc:494] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } I0913 23:13:50.425316 14809 leader_election.cc:296] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3:37091), 93a20b201a904b628926e173bbf23463 (127.13.53.1:44785) I0913 23:13:50.425952 14679 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bf89c16ce260470a863711ddbbdb3d68" candidate_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" is_pre_election: true I0913 23:13:50.426226 14538 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bf89c16ce260470a863711ddbbdb3d68" candidate_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "93a20b201a904b628926e173bbf23463" is_pre_election: true I0913 23:13:50.426869 14538 raft_consensus.cc:2311] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate 0cad5c4a245f4b3dbd66bb7259f45090 for term 3 because replica is either leader or believes a valid leader to be alive. I0913 23:13:50.426910 14679 raft_consensus.cc:2280] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0cad5c4a245f4b3dbd66bb7259f45090 in current term 3: Already voted for candidate 93a20b201a904b628926e173bbf23463 in this term. I0913 23:13:50.427887 14574 leader_election.cc:310] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cad5c4a245f4b3dbd66bb7259f45090; no voters: 93a20b201a904b628926e173bbf23463, a66db810508f4e35bac62f5c6c9f6e9c I0913 23:13:50.428442 14809 raft_consensus.cc:2890] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 2 FOLLOWER]: Advancing to term 3 I0913 23:13:50.432045 14809 raft_consensus.cc:2592] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0913 23:13:50.445574 14679 tablet_service.cc:1440] Received LeaderStepDown RPC: tablet_id: "82f65eba7703468ca650e7ca50d39549" dest_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" from {username='slave'} at 127.0.0.1:59276 I0913 23:13:50.446050 14679 raft_consensus.cc:569] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [term 1 LEADER]: Received request to step down I0913 23:13:50.446331 14679 raft_consensus.cc:2885] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [term 1 LEADER]: Stepping down as leader of term 1 I0913 23:13:50.446476 14679 raft_consensus.cc:705] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a66db810508f4e35bac62f5c6c9f6e9c, State: Running, Role: LEADER I0913 23:13:50.446971 14679 consensus_queue.cc:229] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:50.447805 14679 raft_consensus.cc:2890] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [term 1 FOLLOWER]: Advancing to term 2 I0913 23:13:50.495901 14679 raft_consensus.cc:1184] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [term 3 FOLLOWER]: Refusing update from remote peer 93a20b201a904b628926e173bbf23463: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch) I0913 23:13:50.497114 14821 consensus_queue.cc:984] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I0913 23:13:50.497807 14608 raft_consensus.cc:1184] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 3 FOLLOWER]: Refusing update from remote peer 93a20b201a904b628926e173bbf23463: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch) I0913 23:13:50.499037 14821 consensus_queue.cc:984] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I0913 23:13:50.532470 14821 raft_consensus.cc:472] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a66db810508f4e35bac62f5c6c9f6e9c) I0913 23:13:50.532862 14821 raft_consensus.cc:494] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:50.534154 14821 leader_election.cc:296] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231), a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3:37091) I0913 23:13:50.534819 14608 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "82f65eba7703468ca650e7ca50d39549" candidate_uuid: "93a20b201a904b628926e173bbf23463" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" is_pre_election: true I0913 23:13:50.534991 14679 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "82f65eba7703468ca650e7ca50d39549" candidate_uuid: "93a20b201a904b628926e173bbf23463" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" is_pre_election: true I0913 23:13:50.535584 14608 raft_consensus.cc:2311] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93a20b201a904b628926e173bbf23463 for term 2 because replica is either leader or believes a valid leader to be alive. I0913 23:13:50.535562 14679 raft_consensus.cc:2350] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93a20b201a904b628926e173bbf23463 in term 2. I0913 23:13:50.536420 14496 leader_election.cc:310] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93a20b201a904b628926e173bbf23463, a66db810508f4e35bac62f5c6c9f6e9c; no voters: I0913 23:13:50.537129 14821 raft_consensus.cc:2644] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0913 23:13:50.537384 14821 raft_consensus.cc:472] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a66db810508f4e35bac62f5c6c9f6e9c) I0913 23:13:50.537590 14821 raft_consensus.cc:2890] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [term 1 FOLLOWER]: Advancing to term 2 I0913 23:13:50.541100 14821 raft_consensus.cc:494] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:50.542198 14821 leader_election.cc:296] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [CANDIDATE]: Term 2 election: Requested vote from peers 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231), a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3:37091) I0913 23:13:50.542862 14608 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "82f65eba7703468ca650e7ca50d39549" candidate_uuid: "93a20b201a904b628926e173bbf23463" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" I0913 23:13:50.542919 14679 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "82f65eba7703468ca650e7ca50d39549" candidate_uuid: "93a20b201a904b628926e173bbf23463" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" I0913 23:13:50.543391 14608 raft_consensus.cc:2311] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 93a20b201a904b628926e173bbf23463 for term 2 because replica is either leader or believes a valid leader to be alive. I0913 23:13:50.546720 14679 raft_consensus.cc:2350] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93a20b201a904b628926e173bbf23463 in term 2. I0913 23:13:50.547513 14496 leader_election.cc:310] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93a20b201a904b628926e173bbf23463, a66db810508f4e35bac62f5c6c9f6e9c; no voters: 0cad5c4a245f4b3dbd66bb7259f45090 I0913 23:13:50.548187 14821 raft_consensus.cc:2644] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [term 2 FOLLOWER]: Leader election won for term 2 I0913 23:13:50.548480 14821 raft_consensus.cc:667] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [term 2 LEADER]: Becoming Leader. State: Replica: 93a20b201a904b628926e173bbf23463, State: Running, Role: LEADER I0913 23:13:50.548921 14821 consensus_queue.cc:206] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:50.554141 14441 catalog_manager.cc:4079] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 reported cstate change: term changed from 1 to 2, leader changed from a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3) to 93a20b201a904b628926e173bbf23463 (127.13.53.1). New cstate: current_term: 2 leader_uuid: "93a20b201a904b628926e173bbf23463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } health_report { overall_health: UNKNOWN } } } I0913 23:13:50.610810 14809 raft_consensus.cc:472] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a66db810508f4e35bac62f5c6c9f6e9c) I0913 23:13:50.611266 14809 raft_consensus.cc:494] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:50.612675 14809 leader_election.cc:296] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 93a20b201a904b628926e173bbf23463 (127.13.53.1:44785), a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3:37091) I0913 23:13:50.613348 14538 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "82f65eba7703468ca650e7ca50d39549" candidate_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "93a20b201a904b628926e173bbf23463" is_pre_election: true I0913 23:13:50.613431 14679 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "82f65eba7703468ca650e7ca50d39549" candidate_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" is_pre_election: true I0913 23:13:50.614092 14538 raft_consensus.cc:2311] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 0cad5c4a245f4b3dbd66bb7259f45090 for term 2 because replica is either leader or believes a valid leader to be alive. I0913 23:13:50.614212 14679 raft_consensus.cc:2280] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0cad5c4a245f4b3dbd66bb7259f45090 in current term 2: Already voted for candidate 93a20b201a904b628926e173bbf23463 in this term. I0913 23:13:50.615120 14571 leader_election.cc:310] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cad5c4a245f4b3dbd66bb7259f45090; no voters: 93a20b201a904b628926e173bbf23463, a66db810508f4e35bac62f5c6c9f6e9c I0913 23:13:50.615701 14809 raft_consensus.cc:2890] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 1 FOLLOWER]: Advancing to term 2 I0913 23:13:50.619650 14809 raft_consensus.cc:2592] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0913 23:13:50.628052 14679 raft_consensus.cc:1184] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [term 2 FOLLOWER]: Refusing update from remote peer 93a20b201a904b628926e173bbf23463: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch) I0913 23:13:50.629091 14821 consensus_queue.cc:984] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0913 23:13:50.636752 14608 raft_consensus.cc:1184] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 2 FOLLOWER]: Refusing update from remote peer 93a20b201a904b628926e173bbf23463: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch) I0913 23:13:50.638029 14762 consensus_queue.cc:984] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0913 23:13:50.747512 14439 catalog_manager.cc:2350] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48758: table { table_id: "334306099b944525a47cb78bb2cafd10" table_name: "test-table" } new_table_name: "NewTableName" I0913 23:13:50.773017 14808 raft_consensus.cc:472] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 93a20b201a904b628926e173bbf23463) I0913 23:13:50.773494 14808 raft_consensus.cc:494] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:13:50.777447 14808 leader_election.cc:296] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 93a20b201a904b628926e173bbf23463 (127.13.53.1:44785), 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231) I0913 23:13:50.778964 14538 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8ad9594f28c64b8ba3dacd888b84312b" candidate_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "93a20b201a904b628926e173bbf23463" is_pre_election: true I0913 23:13:50.779723 14538 raft_consensus.cc:2311] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate a66db810508f4e35bac62f5c6c9f6e9c for term 4 because replica is either leader or believes a valid leader to be alive. I0913 23:13:50.781244 14608 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8ad9594f28c64b8ba3dacd888b84312b" candidate_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" is_pre_election: true I0913 23:13:50.781929 14608 raft_consensus.cc:2311] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a66db810508f4e35bac62f5c6c9f6e9c for term 4 because replica is either leader or believes a valid leader to be alive. I0913 23:13:50.783329 14645 leader_election.cc:310] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a66db810508f4e35bac62f5c6c9f6e9c; no voters: 0cad5c4a245f4b3dbd66bb7259f45090, 93a20b201a904b628926e173bbf23463 I0913 23:13:50.786223 14808 raft_consensus.cc:2592] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority I0913 23:13:50.801779 14440 catalog_manager.cc:4174] TS 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231) does not have the latest schema for tablet 82f65eba7703468ca650e7ca50d39549 (table NewTableName [id=334306099b944525a47cb78bb2cafd10]). Expected version 1 got 0 I0913 23:13:50.814965 13524 master.cc:247] Master@127.13.53.60:42667 shutting down... I0913 23:13:50.815443 14439 catalog_manager.cc:4174] TS 93a20b201a904b628926e173bbf23463 (127.13.53.1:44785) does not have the latest schema for tablet 82f65eba7703468ca650e7ca50d39549 (table NewTableName [id=334306099b944525a47cb78bb2cafd10]). Expected version 1 got 0 I0913 23:13:50.899827 14839 tablet.cc:1222] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463: Alter schema from ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0913 23:13:50.900233 14838 tablet.cc:1222] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090: Alter schema from ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0913 23:13:50.905215 14424 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0913 23:13:50.905798 14423 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0913 23:13:50.914295 14833 tablet.cc:1222] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463: Alter schema from ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0913 23:13:50.895159 14835 tablet.cc:1222] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c: Alter schema from ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0913 23:13:50.901084 14836 tablet.cc:1222] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c: Alter schema from ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0913 23:13:50.901293 14834 tablet.cc:1222] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463: Alter schema from ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0913 23:13:50.930822 14424 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0913 23:13:50.936889 14423 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0913 23:13:50.937041 14838 tablet.cc:1222] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090: Alter schema from ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0913 23:13:50.941287 14424 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0913 23:13:50.932152 14837 tablet.cc:1222] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090: Alter schema from ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0913 23:13:50.941685 14840 tablet.cc:1222] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c: Alter schema from ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 10:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0913 23:13:50.953835 14424 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0913 23:13:50.957356 14424 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0913 23:13:50.960068 14839 tablet.cc:1217] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463: Skipping requested alter to schema version 1, tablet already version 1 I0913 23:13:50.965909 14423 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0913 23:13:50.970608 14836 tablet.cc:1217] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c: Skipping requested alter to schema version 1, tablet already version 1 I0913 23:13:50.976481 14424 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0913 23:13:50.980561 14833 tablet.cc:1217] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463: Skipping requested alter to schema version 1, tablet already version 1 I0913 23:13:50.982435 14835 tablet.cc:1217] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c: Skipping requested alter to schema version 1, tablet already version 1 I0913 23:13:50.989795 14423 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0913 23:13:50.990413 14424 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0913 23:13:50.999392 14424 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0913 23:13:51.005175 14837 tablet.cc:1217] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090: Skipping requested alter to schema version 1, tablet already version 1 I0913 23:13:51.006793 14837 tablet.cc:1217] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090: Skipping requested alter to schema version 1, tablet already version 1 I0913 23:13:51.012171 13524 raft_consensus.cc:2147] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 3 LEADER]: Raft consensus shutting down. I0913 23:13:51.013046 13524 raft_consensus.cc:2174] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 3 FOLLOWER]: Raft consensus is shut down! I0913 23:13:51.013442 13524 tablet_replica.cc:273] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607: stopping tablet replica I0913 23:13:51.036862 13524 master.cc:258] Master@127.13.53.60:42667 shutdown complete. I0913 23:13:51.126730 14844 raft_consensus.cc:472] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fab3d6e5bcad48acbb0fe6b928638607) I0913 23:13:51.127162 14844 raft_consensus.cc:494] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:51.128918 14844 leader_election.cc:296] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b03280a8df7e48f9939e076d6d0138ca (127.13.53.61:35977), fab3d6e5bcad48acbb0fe6b928638607 (127.13.53.60:42667) I0913 23:13:51.128882 14845 raft_consensus.cc:472] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fab3d6e5bcad48acbb0fe6b928638607) I0913 23:13:51.129566 14845 raft_consensus.cc:494] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:51.129755 14394 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46574f27d20e427381df2ea44498c90b" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: false dest_uuid: "b03280a8df7e48f9939e076d6d0138ca" is_pre_election: true I0913 23:13:51.130671 14394 raft_consensus.cc:2350] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46574f27d20e427381df2ea44498c90b in term 3. I0913 23:13:51.131945 14310 leader_election.cc:310] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46574f27d20e427381df2ea44498c90b, b03280a8df7e48f9939e076d6d0138ca; no voters: W0913 23:13:51.132656 14311 leader_election.cc:341] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fab3d6e5bcad48acbb0fe6b928638607 (127.13.53.60:42667): Network error: Client connection negotiation failed: client connection to 127.13.53.60:42667: connect: Connection refused (error 111) I0913 23:13:51.132853 14844 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 3 FOLLOWER]: Leader pre-election won for term 4 I0913 23:13:51.133164 14844 raft_consensus.cc:472] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 3 FOLLOWER]: Starting leader election (detected failure of leader fab3d6e5bcad48acbb0fe6b928638607) I0913 23:13:51.133419 14844 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 3 FOLLOWER]: Advancing to term 4 I0913 23:13:51.137533 14844 raft_consensus.cc:494] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:51.138984 14844 leader_election.cc:296] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [CANDIDATE]: Term 4 election: Requested vote from peers b03280a8df7e48f9939e076d6d0138ca (127.13.53.61:35977), fab3d6e5bcad48acbb0fe6b928638607 (127.13.53.60:42667) I0913 23:13:51.139997 14394 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46574f27d20e427381df2ea44498c90b" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: false dest_uuid: "b03280a8df7e48f9939e076d6d0138ca" I0913 23:13:51.140298 14845 leader_election.cc:296] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46574f27d20e427381df2ea44498c90b (127.13.53.62:40439), fab3d6e5bcad48acbb0fe6b928638607 (127.13.53.60:42667) I0913 23:13:51.140596 14394 raft_consensus.cc:2890] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 3 FOLLOWER]: Advancing to term 4 W0913 23:13:51.141005 14311 leader_election.cc:341] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer fab3d6e5bcad48acbb0fe6b928638607 (127.13.53.60:42667): Network error: Client connection negotiation failed: client connection to 127.13.53.60:42667: connect: Connection refused (error 111) I0913 23:13:51.140942 14337 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b03280a8df7e48f9939e076d6d0138ca" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: false dest_uuid: "46574f27d20e427381df2ea44498c90b" is_pre_election: true I0913 23:13:51.141832 14337 raft_consensus.cc:2280] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b03280a8df7e48f9939e076d6d0138ca in current term 4: Already voted for candidate 46574f27d20e427381df2ea44498c90b in this term. W0913 23:13:51.143901 14368 leader_election.cc:341] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fab3d6e5bcad48acbb0fe6b928638607 (127.13.53.60:42667): Network error: Client connection negotiation failed: client connection to 127.13.53.60:42667: connect: Connection refused (error 111) I0913 23:13:51.144188 14368 leader_election.cc:310] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b03280a8df7e48f9939e076d6d0138ca; no voters: 46574f27d20e427381df2ea44498c90b, fab3d6e5bcad48acbb0fe6b928638607 I0913 23:13:51.145810 14394 raft_consensus.cc:2350] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46574f27d20e427381df2ea44498c90b in term 4. I0913 23:13:51.146350 14845 raft_consensus.cc:2592] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority I0913 23:13:51.146966 14310 leader_election.cc:310] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46574f27d20e427381df2ea44498c90b, b03280a8df7e48f9939e076d6d0138ca; no voters: fab3d6e5bcad48acbb0fe6b928638607 I0913 23:13:51.147665 14844 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 4 FOLLOWER]: Leader election won for term 4 I0913 23:13:51.148699 14844 raft_consensus.cc:667] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 4 LEADER]: Becoming Leader. State: Replica: 46574f27d20e427381df2ea44498c90b, State: Running, Role: LEADER I0913 23:13:51.149142 14844 consensus_queue.cc:206] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 3.15, Last appended by leader: 15, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:51.151240 14848 sys_catalog.cc:340] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46574f27d20e427381df2ea44498c90b. Latest consensus state: current_term: 4 leader_uuid: "46574f27d20e427381df2ea44498c90b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:51.151793 14848 sys_catalog.cc:343] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: This master's current role is: LEADER I0913 23:13:51.153393 14849 catalog_manager.cc:1071] Loading table and tablet metadata into memory... I0913 23:13:51.156599 14849 catalog_manager.cc:372] Loaded metadata for table NewTableName [id=334306099b944525a47cb78bb2cafd10] I0913 23:13:51.159421 14849 catalog_manager.cc:426] Loaded metadata for tablet 82f65eba7703468ca650e7ca50d39549 (table NewTableName [id=334306099b944525a47cb78bb2cafd10]) I0913 23:13:51.160037 14849 catalog_manager.cc:426] Loaded metadata for tablet 8ad9594f28c64b8ba3dacd888b84312b (table NewTableName [id=334306099b944525a47cb78bb2cafd10]) I0913 23:13:51.160558 14849 catalog_manager.cc:426] Loaded metadata for tablet bf89c16ce260470a863711ddbbdb3d68 (table NewTableName [id=334306099b944525a47cb78bb2cafd10]) I0913 23:13:51.161170 14849 catalog_manager.cc:1082] Initializing Kudu internal certificate authority... I0913 23:13:51.163920 14849 catalog_manager.cc:1091] Loading token signing keys... I0913 23:13:51.165892 14849 catalog_manager.cc:4405] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b: Loaded TSK: 0 W0913 23:13:51.232545 14311 consensus_peers.cc:458] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b -> Peer fab3d6e5bcad48acbb0fe6b928638607 (127.13.53.60:42667): Couldn't send request to peer fab3d6e5bcad48acbb0fe6b928638607. Status: Network error: Client connection negotiation failed: client connection to 127.13.53.60:42667: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0913 23:13:51.242198 14394 raft_consensus.cc:1184] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 4 FOLLOWER]: Refusing update from remote peer 46574f27d20e427381df2ea44498c90b: Log matching property violated. Preceding OpId in replica: term: 3 index: 15. Preceding OpId from leader: term: 4 index: 16. (index mismatch) I0913 23:13:51.243295 14848 consensus_queue.cc:984] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s I0913 23:13:51.245672 14845 sys_catalog.cc:340] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46574f27d20e427381df2ea44498c90b. Latest consensus state: current_term: 4 leader_uuid: "46574f27d20e427381df2ea44498c90b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:51.246316 14845 sys_catalog.cc:343] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [sys.catalog]: This master's current role is: FOLLOWER I0913 23:13:51.249141 14844 sys_catalog.cc:340] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "46574f27d20e427381df2ea44498c90b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:51.249801 14844 sys_catalog.cc:343] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: This master's current role is: LEADER I0913 23:13:51.249869 14845 sys_catalog.cc:340] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "46574f27d20e427381df2ea44498c90b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:51.250639 14845 sys_catalog.cc:343] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [sys.catalog]: This master's current role is: FOLLOWER I0913 23:13:51.263051 13524 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 20700us I0913 23:13:51.263471 13524 fs_manager.cc:263] Metadata directory not provided I0913 23:13:51.263648 13524 fs_manager.cc:266] Using existing metadata directory in first data directory I0913 23:13:51.266867 13524 fs_manager.cc:399] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s I0913 23:13:51.268349 13524 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0913 23:13:51.277420 13524 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0913 23:13:51.277627 13524 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/master-2-root uuid: "fab3d6e5bcad48acbb0fe6b928638607" format_stamp: "Formatted at 2019-09-13 23:13:47 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:51.277927 13524 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/master-2-root metadata directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/master-2-root 1 data directories: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/master-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0913 23:13:51.309057 14707 heartbeater.cc:447] Master 127.13.53.62:40439 was elected leader, sending a full tablet report... I0913 23:13:51.310256 14636 heartbeater.cc:447] Master 127.13.53.62:40439 was elected leader, sending a full tablet report... I0913 23:13:51.320996 14564 heartbeater.cc:447] Master 127.13.53.62:40439 was elected leader, sending a full tablet report... I0913 23:13:51.328723 14326 catalog_manager.cc:4539] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b: NewTableName [id=334306099b944525a47cb78bb2cafd10] alter complete (version 1) I0913 23:13:51.368432 13524 rpc_server.cc:205] RPC server started. Bound to: 127.13.53.60:42667 I0913 23:13:51.370532 13524 webserver.cc:374] Webserver started at http://127.13.53.60:35131/ using document root and password file I0913 23:13:51.376636 14907 sys_catalog.cc:188] Verifying existing consensus state I0913 23:13:51.381386 14907 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607: Bootstrap starting. I0913 23:13:51.441017 14907 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=0} inserts{seen=6 ignored=0} mutations{seen=12 ignored=0} orphaned_commits=0. Pending: 0 replicates I0913 23:13:51.442186 14907 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607: Bootstrap complete. I0913 23:13:51.446436 14907 raft_consensus.cc:340] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:51.448156 14633 heartbeater.cc:322] Connected to a master server at 127.13.53.60:42667 I0913 23:13:51.449285 14907 raft_consensus.cc:705] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: fab3d6e5bcad48acbb0fe6b928638607, State: Initialized, Role: FOLLOWER I0913 23:13:51.450022 14907 consensus_queue.cc:229] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 3.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:51.452962 14704 heartbeater.cc:322] Connected to a master server at 127.13.53.60:42667 I0913 23:13:51.456970 14907 sys_catalog.cc:340] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:51.457993 14907 sys_catalog.cc:343] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: This master's current role is: FOLLOWER I0913 23:13:51.457152 14913 sys_catalog.cc:340] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:51.458932 14913 sys_catalog.cc:343] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: This master's current role is: FOLLOWER I0913 23:13:51.458987 14907 sys_catalog.cc:428] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: configured and running, proceeding with master startup. I0913 23:13:51.459327 14884 raft_consensus.cc:2890] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 3 FOLLOWER]: Advancing to term 4 I0913 23:13:51.464965 14562 heartbeater.cc:322] Connected to a master server at 127.13.53.60:42667 I0913 23:13:51.476009 14874 master_service.cc:230] Got heartbeat from unknown tserver (permanent_uuid: "93a20b201a904b628926e173bbf23463" instance_seqno: 1568416427412171) as {username='slave'} at 127.0.0.1:48796; Asking this server to re-register. I0913 23:13:51.476032 14913 sys_catalog.cc:340] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46574f27d20e427381df2ea44498c90b. Latest consensus state: current_term: 4 leader_uuid: "46574f27d20e427381df2ea44498c90b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:51.476776 14913 sys_catalog.cc:343] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: This master's current role is: FOLLOWER I0913 23:13:51.477267 14917 sys_catalog.cc:340] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "46574f27d20e427381df2ea44498c90b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:51.477895 14917 sys_catalog.cc:343] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: This master's current role is: FOLLOWER I0913 23:13:51.477880 14562 heartbeater.cc:406] Registering TS with master... I0913 23:13:51.479161 13524 master.cc:247] Master@127.13.53.62:40439 shutting down... I0913 23:13:51.484186 14309 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0913 23:13:51.489404 14311 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0913 23:13:51.490633 14309 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0913 23:13:51.492072 14310 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0913 23:13:51.493999 14874 ts_manager.cc:161] Registered new tserver with Master: 93a20b201a904b628926e173bbf23463 (127.13.53.1:44785) I0913 23:13:51.494467 14914 catalog_manager.cc:1127] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607: acquiring CA information for follower catalog manager: success I0913 23:13:51.498751 14914 catalog_manager.cc:1155] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0913 23:13:51.499362 14311 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0913 23:13:51.500572 14873 master_service.cc:230] Got heartbeat from unknown tserver (permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" instance_seqno: 1568416427733869) as {username='slave'} at 127.0.0.1:48794; Asking this server to re-register. I0913 23:13:51.502261 14704 heartbeater.cc:406] Registering TS with master... I0913 23:13:51.507102 14872 ts_manager.cc:161] Registered new tserver with Master: a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3:37091) I0913 23:13:51.506896 14850 sys_catalog.cc:340] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "46574f27d20e427381df2ea44498c90b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:51.507694 14850 sys_catalog.cc:343] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: This master's current role is: LEADER I0913 23:13:51.507668 14874 master_service.cc:230] Got heartbeat from unknown tserver (permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" instance_seqno: 1568416427553058) as {username='slave'} at 127.0.0.1:48790; Asking this server to re-register. I0913 23:13:51.510670 14310 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0913 23:13:51.513643 14633 heartbeater.cc:406] Registering TS with master... I0913 23:13:51.516963 14874 ts_manager.cc:161] Registered new tserver with Master: 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231) I0913 23:13:51.531074 13524 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 4 LEADER]: Raft consensus shutting down. I0913 23:13:51.531807 13524 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 4 FOLLOWER]: Raft consensus is shut down! I0913 23:13:51.532186 13524 tablet_replica.cc:273] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b: stopping tablet replica I0913 23:13:51.552657 13524 master.cc:258] Master@127.13.53.62:40439 shutdown complete. I0913 23:13:51.629011 14845 raft_consensus.cc:472] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46574f27d20e427381df2ea44498c90b) I0913 23:13:51.629446 14845 raft_consensus.cc:494] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:51.630708 14845 leader_election.cc:296] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46574f27d20e427381df2ea44498c90b (127.13.53.62:40439), fab3d6e5bcad48acbb0fe6b928638607 (127.13.53.60:42667) W0913 23:13:51.632086 14366 leader_election.cc:341] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 46574f27d20e427381df2ea44498c90b (127.13.53.62:40439): Network error: Client connection negotiation failed: client connection to 127.13.53.62:40439: connect: Connection refused (error 111) I0913 23:13:51.640105 14884 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b03280a8df7e48f9939e076d6d0138ca" candidate_term: 5 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "fab3d6e5bcad48acbb0fe6b928638607" is_pre_election: true I0913 23:13:51.640658 14884 raft_consensus.cc:2311] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b03280a8df7e48f9939e076d6d0138ca for term 5 because replica is either leader or believes a valid leader to be alive. I0913 23:13:51.641403 14368 leader_election.cc:310] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b03280a8df7e48f9939e076d6d0138ca; no voters: 46574f27d20e427381df2ea44498c90b, fab3d6e5bcad48acbb0fe6b928638607 I0913 23:13:51.642053 14845 raft_consensus.cc:2592] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I0913 23:13:51.707818 14917 raft_consensus.cc:472] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46574f27d20e427381df2ea44498c90b) I0913 23:13:51.708302 14917 raft_consensus.cc:494] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } W0913 23:13:51.709340 14575 debug-util.cc:397] Leaking SignalData structure 0x7b080013d7a0 after lost signal to thread 13527 I0913 23:13:51.709923 14917 leader_election.cc:296] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46574f27d20e427381df2ea44498c90b (127.13.53.62:40439), b03280a8df7e48f9939e076d6d0138ca (127.13.53.61:35977) W0913 23:13:51.713328 14575 debug-util.cc:397] Leaking SignalData structure 0x7b080011ac80 after lost signal to thread 14631 W0913 23:13:51.714500 14575 debug-util.cc:397] Leaking SignalData structure 0x7b08000e61c0 after lost signal to thread 14702 W0913 23:13:51.726922 14854 leader_election.cc:341] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 46574f27d20e427381df2ea44498c90b (127.13.53.62:40439): Network error: Client connection negotiation failed: client connection to 127.13.53.62:40439: connect: Connection refused (error 111) I0913 23:13:51.733577 14394 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fab3d6e5bcad48acbb0fe6b928638607" candidate_term: 5 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "b03280a8df7e48f9939e076d6d0138ca" is_pre_election: true I0913 23:13:51.734315 14394 raft_consensus.cc:2350] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fab3d6e5bcad48acbb0fe6b928638607 in term 4. I0913 23:13:51.735232 14855 leader_election.cc:310] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b03280a8df7e48f9939e076d6d0138ca, fab3d6e5bcad48acbb0fe6b928638607; no voters: 46574f27d20e427381df2ea44498c90b I0913 23:13:51.736065 14917 raft_consensus.cc:2644] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 4 FOLLOWER]: Leader pre-election won for term 5 I0913 23:13:51.736366 14917 raft_consensus.cc:472] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 46574f27d20e427381df2ea44498c90b) I0913 23:13:51.736590 14917 raft_consensus.cc:2890] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 4 FOLLOWER]: Advancing to term 5 I0913 23:13:51.740509 14917 raft_consensus.cc:494] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:51.741683 14917 leader_election.cc:296] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [CANDIDATE]: Term 5 election: Requested vote from peers 46574f27d20e427381df2ea44498c90b (127.13.53.62:40439), b03280a8df7e48f9939e076d6d0138ca (127.13.53.61:35977) I0913 23:13:51.742628 14394 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fab3d6e5bcad48acbb0fe6b928638607" candidate_term: 5 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "b03280a8df7e48f9939e076d6d0138ca" W0913 23:13:51.742910 14854 leader_election.cc:341] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 46574f27d20e427381df2ea44498c90b (127.13.53.62:40439): Network error: Client connection negotiation failed: client connection to 127.13.53.62:40439: connect: Connection refused (error 111) I0913 23:13:51.743185 14394 raft_consensus.cc:2890] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 4 FOLLOWER]: Advancing to term 5 I0913 23:13:51.746954 14394 raft_consensus.cc:2350] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fab3d6e5bcad48acbb0fe6b928638607 in term 5. I0913 23:13:51.747735 14855 leader_election.cc:310] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b03280a8df7e48f9939e076d6d0138ca, fab3d6e5bcad48acbb0fe6b928638607; no voters: 46574f27d20e427381df2ea44498c90b I0913 23:13:51.748332 14917 raft_consensus.cc:2644] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 5 FOLLOWER]: Leader election won for term 5 I0913 23:13:51.748684 14917 raft_consensus.cc:667] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 5 LEADER]: Becoming Leader. State: Replica: fab3d6e5bcad48acbb0fe6b928638607, State: Running, Role: LEADER I0913 23:13:51.749150 14917 consensus_queue.cc:206] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 4.17, Last appended by leader: 17, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:51.750814 14913 sys_catalog.cc:340] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fab3d6e5bcad48acbb0fe6b928638607. Latest consensus state: current_term: 5 leader_uuid: "fab3d6e5bcad48acbb0fe6b928638607" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:51.751363 14913 sys_catalog.cc:343] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: This master's current role is: LEADER I0913 23:13:51.753010 14923 catalog_manager.cc:1071] Loading table and tablet metadata into memory... I0913 23:13:51.756752 14923 catalog_manager.cc:372] Loaded metadata for table NewTableName [id=334306099b944525a47cb78bb2cafd10] I0913 23:13:51.760339 14923 catalog_manager.cc:426] Loaded metadata for tablet 82f65eba7703468ca650e7ca50d39549 (table NewTableName [id=334306099b944525a47cb78bb2cafd10]) I0913 23:13:51.761171 14923 catalog_manager.cc:426] Loaded metadata for tablet 8ad9594f28c64b8ba3dacd888b84312b (table NewTableName [id=334306099b944525a47cb78bb2cafd10]) I0913 23:13:51.762001 14923 catalog_manager.cc:426] Loaded metadata for tablet bf89c16ce260470a863711ddbbdb3d68 (table NewTableName [id=334306099b944525a47cb78bb2cafd10]) I0913 23:13:51.762733 14923 catalog_manager.cc:1082] Initializing Kudu internal certificate authority... I0913 23:13:51.766599 14923 catalog_manager.cc:1091] Loading token signing keys... I0913 23:13:51.769542 14923 catalog_manager.cc:4405] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607: Loaded TSK: 0 I0913 23:13:51.782197 13524 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 20700us I0913 23:13:51.782701 13524 fs_manager.cc:263] Metadata directory not provided I0913 23:13:51.782884 13524 fs_manager.cc:266] Using existing metadata directory in first data directory I0913 23:13:51.785960 13524 fs_manager.cc:399] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s I0913 23:13:51.787427 13524 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0913 23:13:51.800333 13524 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s I0913 23:13:51.800622 13524 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/master-0-root uuid: "46574f27d20e427381df2ea44498c90b" format_stamp: "Formatted at 2019-09-13 23:13:46 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:51.800943 13524 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/master-0-root metadata directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0913 23:13:51.849541 14394 raft_consensus.cc:1184] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 5 FOLLOWER]: Refusing update from remote peer fab3d6e5bcad48acbb0fe6b928638607: Log matching property violated. Preceding OpId in replica: term: 4 index: 17. Preceding OpId from leader: term: 5 index: 18. (index mismatch) I0913 23:13:51.851219 14913 consensus_queue.cc:984] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s I0913 23:13:51.854696 14845 sys_catalog.cc:340] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader fab3d6e5bcad48acbb0fe6b928638607. Latest consensus state: current_term: 5 leader_uuid: "fab3d6e5bcad48acbb0fe6b928638607" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:51.855509 14845 sys_catalog.cc:343] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [sys.catalog]: This master's current role is: FOLLOWER I0913 23:13:51.859572 14917 sys_catalog.cc:340] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "fab3d6e5bcad48acbb0fe6b928638607" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:51.860936 14917 sys_catalog.cc:343] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: This master's current role is: LEADER W0913 23:13:51.860839 14854 consensus_peers.cc:458] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 -> Peer 46574f27d20e427381df2ea44498c90b (127.13.53.62:40439): Couldn't send request to peer 46574f27d20e427381df2ea44498c90b. Status: Network error: Client connection negotiation failed: client connection to 127.13.53.62:40439: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0913 23:13:51.870254 14845 sys_catalog.cc:340] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "fab3d6e5bcad48acbb0fe6b928638607" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:51.871093 14845 sys_catalog.cc:343] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [sys.catalog]: This master's current role is: FOLLOWER I0913 23:13:51.916263 13524 rpc_server.cc:205] RPC server started. Bound to: 127.13.53.62:40439 I0913 23:13:51.918990 13524 webserver.cc:374] Webserver started at http://127.13.53.62:33569/ using document root and password file I0913 23:13:51.922971 14704 heartbeater.cc:447] Master 127.13.53.60:42667 was elected leader, sending a full tablet report... I0913 23:13:51.927860 14980 sys_catalog.cc:188] Verifying existing consensus state I0913 23:13:51.928576 14562 heartbeater.cc:447] Master 127.13.53.60:42667 was elected leader, sending a full tablet report... I0913 23:13:51.932307 14633 heartbeater.cc:447] Master 127.13.53.60:42667 was elected leader, sending a full tablet report... I0913 23:13:51.942647 14980 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b: Bootstrap starting. I0913 23:13:51.945597 14707 heartbeater.cc:322] Connected to a master server at 127.13.53.62:40439 I0913 23:13:51.992347 14980 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=1 applied=17 ignored=0} inserts{seen=6 ignored=0} mutations{seen=13 ignored=0} orphaned_commits=0. Pending: 0 replicates I0913 23:13:51.993333 14980 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b: Bootstrap complete. I0913 23:13:51.996886 14980 raft_consensus.cc:340] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 4 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:51.997658 14980 raft_consensus.cc:705] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46574f27d20e427381df2ea44498c90b, State: Initialized, Role: FOLLOWER I0913 23:13:51.998275 14980 consensus_queue.cc:229] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 17, Last appended: 4.17, Last appended by leader: 17, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:52.000205 14980 sys_catalog.cc:340] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:52.000452 14983 sys_catalog.cc:340] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:52.000969 14980 sys_catalog.cc:343] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: This master's current role is: FOLLOWER I0913 23:13:52.001202 14983 sys_catalog.cc:343] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: This master's current role is: FOLLOWER I0913 23:13:52.002218 14980 sys_catalog.cc:428] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: configured and running, proceeding with master startup. I0913 23:13:52.002542 14957 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 4 FOLLOWER]: Advancing to term 5 I0913 23:13:52.017705 14983 sys_catalog.cc:340] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fab3d6e5bcad48acbb0fe6b928638607. Latest consensus state: current_term: 5 leader_uuid: "fab3d6e5bcad48acbb0fe6b928638607" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:52.018754 14983 sys_catalog.cc:343] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: This master's current role is: FOLLOWER I0913 23:13:52.019188 14983 sys_catalog.cc:340] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "fab3d6e5bcad48acbb0fe6b928638607" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:52.019830 14983 sys_catalog.cc:343] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: This master's current role is: FOLLOWER I0913 23:13:52.031664 13524 master.cc:247] Master@127.13.53.60:42667 shutting down... I0913 23:13:52.032580 14984 catalog_manager.cc:1127] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b: acquiring CA information for follower catalog manager: success I0913 23:13:52.035085 14947 master_service.cc:230] Got heartbeat from unknown tserver (permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" instance_seqno: 1568416427733869) as {username='slave'} at 127.0.0.1:42246; Asking this server to re-register. I0913 23:13:52.036774 14984 catalog_manager.cc:1155] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0913 23:13:52.036960 14564 heartbeater.cc:322] Connected to a master server at 127.13.53.62:40439 I0913 23:13:52.037705 14707 heartbeater.cc:406] Registering TS with master... I0913 23:13:52.040758 14947 master_service.cc:230] Got heartbeat from unknown tserver (permanent_uuid: "93a20b201a904b628926e173bbf23463" instance_seqno: 1568416427412171) as {username='slave'} at 127.0.0.1:42250; Asking this server to re-register. I0913 23:13:52.042021 14946 ts_manager.cc:161] Registered new tserver with Master: a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3:37091) I0913 23:13:52.042156 14564 heartbeater.cc:406] Registering TS with master... I0913 23:13:52.045094 14946 ts_manager.cc:161] Registered new tserver with Master: 93a20b201a904b628926e173bbf23463 (127.13.53.1:44785) I0913 23:13:52.046738 14856 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0913 23:13:52.053105 14856 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0913 23:13:52.073472 14917 sys_catalog.cc:340] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "fab3d6e5bcad48acbb0fe6b928638607" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:52.074905 14917 sys_catalog.cc:343] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: This master's current role is: LEADER I0913 23:13:52.076268 14636 heartbeater.cc:322] Connected to a master server at 127.13.53.62:40439 I0913 23:13:52.078886 14946 master_service.cc:230] Got heartbeat from unknown tserver (permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" instance_seqno: 1568416427553058) as {username='slave'} at 127.0.0.1:42252; Asking this server to re-register. I0913 23:13:52.081513 14636 heartbeater.cc:406] Registering TS with master... I0913 23:13:52.086539 14946 ts_manager.cc:161] Registered new tserver with Master: 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231) I0913 23:13:52.091864 13524 raft_consensus.cc:2147] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 5 LEADER]: Raft consensus shutting down. I0913 23:13:52.092785 13524 raft_consensus.cc:2174] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 5 FOLLOWER]: Raft consensus is shut down! I0913 23:13:52.093494 13524 tablet_replica.cc:273] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607: stopping tablet replica I0913 23:13:52.107319 13524 master.cc:258] Master@127.13.53.60:42667 shutdown complete. I0913 23:13:52.255976 14845 raft_consensus.cc:472] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fab3d6e5bcad48acbb0fe6b928638607) I0913 23:13:52.256376 14845 raft_consensus.cc:494] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:52.257771 14845 leader_election.cc:296] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46574f27d20e427381df2ea44498c90b (127.13.53.62:40439), fab3d6e5bcad48acbb0fe6b928638607 (127.13.53.60:42667) W0913 23:13:52.267377 14368 leader_election.cc:341] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer fab3d6e5bcad48acbb0fe6b928638607 (127.13.53.60:42667): Network error: Client connection negotiation failed: client connection to 127.13.53.60:42667: connect: Connection refused (error 111) I0913 23:13:52.276600 14957 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b03280a8df7e48f9939e076d6d0138ca" candidate_term: 6 candidate_status { last_received { term: 5 index: 18 } } ignore_live_leader: false dest_uuid: "46574f27d20e427381df2ea44498c90b" is_pre_election: true I0913 23:13:52.277485 14957 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b03280a8df7e48f9939e076d6d0138ca in term 5. I0913 23:13:52.277899 14988 raft_consensus.cc:472] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fab3d6e5bcad48acbb0fe6b928638607) I0913 23:13:52.278328 14988 raft_consensus.cc:494] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:52.278550 14366 leader_election.cc:310] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46574f27d20e427381df2ea44498c90b, b03280a8df7e48f9939e076d6d0138ca; no voters: fab3d6e5bcad48acbb0fe6b928638607 I0913 23:13:52.279409 14845 raft_consensus.cc:2644] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 5 FOLLOWER]: Leader pre-election won for term 6 I0913 23:13:52.279702 14845 raft_consensus.cc:472] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 5 FOLLOWER]: Starting leader election (detected failure of leader fab3d6e5bcad48acbb0fe6b928638607) I0913 23:13:52.279700 14988 leader_election.cc:296] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b03280a8df7e48f9939e076d6d0138ca (127.13.53.61:35977), fab3d6e5bcad48acbb0fe6b928638607 (127.13.53.60:42667) I0913 23:13:52.280117 14845 raft_consensus.cc:2890] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 5 FOLLOWER]: Advancing to term 6 I0913 23:13:52.284143 14845 raft_consensus.cc:494] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:52.285334 14845 leader_election.cc:296] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [CANDIDATE]: Term 6 election: Requested vote from peers 46574f27d20e427381df2ea44498c90b (127.13.53.62:40439), fab3d6e5bcad48acbb0fe6b928638607 (127.13.53.60:42667) W0913 23:13:52.290114 14368 leader_election.cc:341] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer fab3d6e5bcad48acbb0fe6b928638607 (127.13.53.60:42667): Network error: Client connection negotiation failed: client connection to 127.13.53.60:42667: connect: Connection refused (error 111) I0913 23:13:52.291702 14957 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b03280a8df7e48f9939e076d6d0138ca" candidate_term: 6 candidate_status { last_received { term: 5 index: 18 } } ignore_live_leader: false dest_uuid: "46574f27d20e427381df2ea44498c90b" I0913 23:13:52.292410 14957 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 5 FOLLOWER]: Advancing to term 6 I0913 23:13:52.303023 14957 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b03280a8df7e48f9939e076d6d0138ca in term 6. I0913 23:13:52.305554 14366 leader_election.cc:310] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46574f27d20e427381df2ea44498c90b, b03280a8df7e48f9939e076d6d0138ca; no voters: fab3d6e5bcad48acbb0fe6b928638607 I0913 23:13:52.306236 14845 raft_consensus.cc:2644] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 6 FOLLOWER]: Leader election won for term 6 W0913 23:13:52.306690 14930 leader_election.cc:341] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer fab3d6e5bcad48acbb0fe6b928638607 (127.13.53.60:42667): Network error: Client connection negotiation failed: client connection to 127.13.53.60:42667: connect: Connection refused (error 111) I0913 23:13:52.309603 14394 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46574f27d20e427381df2ea44498c90b" candidate_term: 6 candidate_status { last_received { term: 5 index: 18 } } ignore_live_leader: false dest_uuid: "b03280a8df7e48f9939e076d6d0138ca" is_pre_election: true I0913 23:13:52.310483 14845 raft_consensus.cc:667] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 6 LEADER]: Becoming Leader. State: Replica: b03280a8df7e48f9939e076d6d0138ca, State: Running, Role: LEADER I0913 23:13:52.310958 14845 consensus_queue.cc:206] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 5.18, Last appended by leader: 18, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:52.312659 14394 raft_consensus.cc:2311] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 6 LEADER]: Leader pre-election vote request: Denying vote to candidate 46574f27d20e427381df2ea44498c90b for term 6 because replica is either leader or believes a valid leader to be alive. I0913 23:13:52.314563 14929 leader_election.cc:310] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46574f27d20e427381df2ea44498c90b; no voters: b03280a8df7e48f9939e076d6d0138ca, fab3d6e5bcad48acbb0fe6b928638607 I0913 23:13:52.315357 14988 raft_consensus.cc:2592] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0913 23:13:52.313678 14995 sys_catalog.cc:340] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader b03280a8df7e48f9939e076d6d0138ca. Latest consensus state: current_term: 6 leader_uuid: "b03280a8df7e48f9939e076d6d0138ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:52.322556 14995 sys_catalog.cc:343] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [sys.catalog]: This master's current role is: LEADER I0913 23:13:52.330727 14996 catalog_manager.cc:1071] Loading table and tablet metadata into memory... I0913 23:13:52.334307 14996 catalog_manager.cc:372] Loaded metadata for table NewTableName [id=334306099b944525a47cb78bb2cafd10] I0913 23:13:52.336761 14996 catalog_manager.cc:426] Loaded metadata for tablet 82f65eba7703468ca650e7ca50d39549 (table NewTableName [id=334306099b944525a47cb78bb2cafd10]) I0913 23:13:52.337440 14996 catalog_manager.cc:426] Loaded metadata for tablet 8ad9594f28c64b8ba3dacd888b84312b (table NewTableName [id=334306099b944525a47cb78bb2cafd10]) I0913 23:13:52.338155 14996 catalog_manager.cc:426] Loaded metadata for tablet bf89c16ce260470a863711ddbbdb3d68 (table NewTableName [id=334306099b944525a47cb78bb2cafd10]) I0913 23:13:52.338917 14996 catalog_manager.cc:1082] Initializing Kudu internal certificate authority... I0913 23:13:52.339329 13524 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 21200us I0913 23:13:52.339676 13524 fs_manager.cc:263] Metadata directory not provided I0913 23:13:52.339829 13524 fs_manager.cc:266] Using existing metadata directory in first data directory I0913 23:13:52.341917 14996 catalog_manager.cc:1091] Loading token signing keys... I0913 23:13:52.344667 14996 catalog_manager.cc:4405] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca: Loaded TSK: 0 I0913 23:13:52.347242 13524 fs_manager.cc:399] Time spent opening directory manager: real 0.007s user 0.002s sys 0.001s I0913 23:13:52.348799 13524 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0913 23:13:52.363423 13524 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s I0913 23:13:52.363696 13524 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/master-2-root uuid: "fab3d6e5bcad48acbb0fe6b928638607" format_stamp: "Formatted at 2019-09-13 23:13:47 on dist-test-slave-dist-test-slave-kz3c" I0913 23:13:52.364006 13524 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/master-2-root metadata directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/master-2-root 1 data directories: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524/minicluster-data/master-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0913 23:13:52.418380 14368 consensus_peers.cc:458] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca -> Peer fab3d6e5bcad48acbb0fe6b928638607 (127.13.53.60:42667): Couldn't send request to peer fab3d6e5bcad48acbb0fe6b928638607. Status: Network error: Client connection negotiation failed: client connection to 127.13.53.60:42667: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0913 23:13:52.429669 14957 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 6 FOLLOWER]: Refusing update from remote peer b03280a8df7e48f9939e076d6d0138ca: Log matching property violated. Preceding OpId in replica: term: 5 index: 18. Preceding OpId from leader: term: 6 index: 19. (index mismatch) I0913 23:13:52.431788 14995 consensus_queue.cc:984] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s I0913 23:13:52.436311 14988 sys_catalog.cc:340] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b03280a8df7e48f9939e076d6d0138ca. Latest consensus state: current_term: 6 leader_uuid: "b03280a8df7e48f9939e076d6d0138ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:52.437134 14988 sys_catalog.cc:343] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: This master's current role is: FOLLOWER I0913 23:13:52.447958 14988 sys_catalog.cc:340] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "b03280a8df7e48f9939e076d6d0138ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:52.448752 14988 sys_catalog.cc:343] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [sys.catalog]: This master's current role is: FOLLOWER I0913 23:13:52.458427 14563 heartbeater.cc:447] Master 127.13.53.61:35977 was elected leader, sending a full tablet report... I0913 23:13:52.460762 15016 sys_catalog.cc:340] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "b03280a8df7e48f9939e076d6d0138ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:52.461591 15016 sys_catalog.cc:343] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [sys.catalog]: This master's current role is: LEADER I0913 23:13:52.473652 14705 heartbeater.cc:447] Master 127.13.53.61:35977 was elected leader, sending a full tablet report... I0913 23:13:52.500905 13524 rpc_server.cc:205] RPC server started. Bound to: 127.13.53.60:42667 I0913 23:13:52.503271 13524 webserver.cc:374] Webserver started at http://127.13.53.60:35131/ using document root and password file I0913 23:13:52.509096 15053 sys_catalog.cc:188] Verifying existing consensus state I0913 23:13:52.513932 15053 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607: Bootstrap starting. I0913 23:13:52.550581 14704 heartbeater.cc:322] Connected to a master server at 127.13.53.60:42667 I0913 23:13:52.553381 15053 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=6 ignored=0} mutations{seen=13 ignored=0} orphaned_commits=0. Pending: 0 replicates I0913 23:13:52.554427 15053 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607: Bootstrap complete. I0913 23:13:52.556797 15053 raft_consensus.cc:340] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:52.557565 15053 raft_consensus.cc:705] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: fab3d6e5bcad48acbb0fe6b928638607, State: Initialized, Role: FOLLOWER I0913 23:13:52.558218 15053 consensus_queue.cc:229] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 5.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } I0913 23:13:52.560622 14634 heartbeater.cc:447] Master 127.13.53.61:35977 was elected leader, sending a full tablet report... I0913 23:13:52.567581 15053 sys_catalog.cc:340] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:52.568403 15053 sys_catalog.cc:343] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: This master's current role is: FOLLOWER I0913 23:13:52.570910 15053 sys_catalog.cc:428] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: configured and running, proceeding with master startup. I0913 23:13:52.571017 15030 raft_consensus.cc:2890] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 5 FOLLOWER]: Advancing to term 6 I0913 23:13:52.573266 14633 heartbeater.cc:322] Connected to a master server at 127.13.53.60:42667 I0913 23:13:52.570869 15056 sys_catalog.cc:340] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:52.576084 15056 sys_catalog.cc:343] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: This master's current role is: FOLLOWER I0913 23:13:52.584244 14562 heartbeater.cc:322] Connected to a master server at 127.13.53.60:42667 I0913 23:13:52.586803 15056 sys_catalog.cc:340] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b03280a8df7e48f9939e076d6d0138ca. Latest consensus state: current_term: 6 leader_uuid: "b03280a8df7e48f9939e076d6d0138ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:52.587525 15056 sys_catalog.cc:343] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: This master's current role is: FOLLOWER I0913 23:13:52.588558 15020 master_service.cc:230] Got heartbeat from unknown tserver (permanent_uuid: "93a20b201a904b628926e173bbf23463" instance_seqno: 1568416427412171) as {username='slave'} at 127.0.0.1:48882; Asking this server to re-register. I0913 23:13:52.590039 14562 heartbeater.cc:406] Registering TS with master... I0913 23:13:52.590267 15061 sys_catalog.cc:340] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "b03280a8df7e48f9939e076d6d0138ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:52.590896 15061 sys_catalog.cc:343] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [sys.catalog]: This master's current role is: FOLLOWER I0913 23:13:52.593019 15059 catalog_manager.cc:1127] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607: acquiring CA information for follower catalog manager: success I0913 23:13:52.593179 15020 ts_manager.cc:161] Registered new tserver with Master: 93a20b201a904b628926e173bbf23463 (127.13.53.1:44785) I0913 23:13:52.596815 15059 catalog_manager.cc:1155] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0913 23:13:52.609740 15020 master_service.cc:230] Got heartbeat from unknown tserver (permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" instance_seqno: 1568416427553058) as {username='slave'} at 127.0.0.1:48880; Asking this server to re-register. I0913 23:13:52.611331 14633 heartbeater.cc:406] Registering TS with master... I0913 23:13:52.616123 15019 ts_manager.cc:161] Registered new tserver with Master: 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231) I0913 23:13:52.618461 14845 sys_catalog.cc:340] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "b03280a8df7e48f9939e076d6d0138ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46574f27d20e427381df2ea44498c90b" member_type: VOTER last_known_addr { host: "127.13.53.62" port: 40439 } } peers { permanent_uuid: "b03280a8df7e48f9939e076d6d0138ca" member_type: VOTER last_known_addr { host: "127.13.53.61" port: 35977 } } peers { permanent_uuid: "fab3d6e5bcad48acbb0fe6b928638607" member_type: VOTER last_known_addr { host: "127.13.53.60" port: 42667 } } } I0913 23:13:52.619227 14845 sys_catalog.cc:343] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [sys.catalog]: This master's current role is: LEADER I0913 23:13:52.624914 15020 master_service.cc:230] Got heartbeat from unknown tserver (permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" instance_seqno: 1568416427733869) as {username='slave'} at 127.0.0.1:48878; Asking this server to re-register. I0913 23:13:52.626179 14704 heartbeater.cc:406] Registering TS with master... I0913 23:13:52.628525 15020 ts_manager.cc:161] Registered new tserver with Master: a66db810508f4e35bac62f5c6c9f6e9c (127.13.53.3:37091) W0913 23:13:59.779052 14931 debug-util.cc:397] Leaking SignalData structure 0x7b08001f2a00 after lost signal to thread 13527 W0913 23:13:59.779927 14931 debug-util.cc:397] Leaking SignalData structure 0x7b08000d8140 after lost signal to thread 14415 W0913 23:13:59.780647 14931 debug-util.cc:397] Leaking SignalData structure 0x7b08000d8980 after lost signal to thread 14560 W0913 23:13:59.781352 14931 debug-util.cc:397] Leaking SignalData structure 0x7b08001f4a40 after lost signal to thread 14631 W0913 23:13:59.782110 14931 debug-util.cc:397] Leaking SignalData structure 0x7b0800225580 after lost signal to thread 14702 W0913 23:13:59.782799 14931 debug-util.cc:397] Leaking SignalData structure 0x7b08000bed00 after lost signal to thread 14978 W0913 23:13:59.783418 14931 debug-util.cc:397] Leaking SignalData structure 0x7b08001ec880 after lost signal to thread 15051 /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/ts_tablet_manager-itest.cc:642: Failure Expected: live_row_count Which is: 327 To be equal to: table_info->GetMetrics()->live_row_count->value() Which is: 654 /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/test_util.cc:326: Failure Failed Timed out waiting for assertion to pass. /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/ts_tablet_manager-itest.cc:627: Failure Expected: check_function(table_infos[0].get(), live_row_count) doesn't generate new fatal failures in the current thread. Actual: it does. /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/ts_tablet_manager-itest.cc:644: Failure Expected: GetLeaderMasterAndRun(live_row_count, [&] ( TableInfo* table_info, int64_t live_row_count) { do { AssertEventually([&] () { switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar = (::testing::internal:: EqHelper<(sizeof(::testing::internal::IsNullLiteralHelper(live_row_count)) == 1)>::Compare("live_row_count", "table_info->GetMetrics()->live_row_count->value()", live_row_count, table_info->GetMetrics()->live_row_count->value()))) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 642, gtest_ar.failure_message()) = ::testing::Message(); }); if (testing::Test::HasFatalFailure()) { return; }; } while (0); }) doesn't generate new fatal failures in the current thread. Actual: it does. /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/ts_tablet_manager-itest.cc:742: Failure Expected: CheckStats(kRowsCount) doesn't generate new fatal failures in the current thread. Actual: it does. I0913 23:14:22.723248 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:14:22.759454 13524 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0913 23:14:22.759810 13524 tablet_replica.cc:273] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463: stopping tablet replica I0913 23:14:22.760180 13524 raft_consensus.cc:2147] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [term 2 LEADER]: Raft consensus shutting down. I0913 23:14:22.760841 13524 raft_consensus.cc:2174] T 82f65eba7703468ca650e7ca50d39549 P 93a20b201a904b628926e173bbf23463 [term 2 FOLLOWER]: Raft consensus is shut down! I0913 23:14:22.763161 13524 tablet_replica.cc:273] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463: stopping tablet replica I0913 23:14:22.763480 13524 raft_consensus.cc:2147] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [term 3 LEADER]: Raft consensus shutting down. I0913 23:14:22.764199 13524 raft_consensus.cc:2174] T bf89c16ce260470a863711ddbbdb3d68 P 93a20b201a904b628926e173bbf23463 [term 3 FOLLOWER]: Raft consensus is shut down! I0913 23:14:22.766422 13524 tablet_replica.cc:273] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463: stopping tablet replica I0913 23:14:22.766711 13524 raft_consensus.cc:2147] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [term 3 LEADER]: Raft consensus shutting down. I0913 23:14:22.767272 13524 raft_consensus.cc:2174] T 8ad9594f28c64b8ba3dacd888b84312b P 93a20b201a904b628926e173bbf23463 [term 3 FOLLOWER]: Raft consensus is shut down! I0913 23:14:22.788918 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:14:22.789129 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:14:22.789412 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:14:22.800873 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:14:22.831351 13524 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0913 23:14:22.831697 13524 tablet_replica.cc:273] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090: stopping tablet replica I0913 23:14:22.832036 13524 raft_consensus.cc:2147] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [term 3 FOLLOWER]: Raft consensus shutting down. I0913 23:14:22.832286 13524 raft_consensus.cc:2174] T 8ad9594f28c64b8ba3dacd888b84312b P 0cad5c4a245f4b3dbd66bb7259f45090 [term 3 FOLLOWER]: Raft consensus is shut down! I0913 23:14:22.834314 13524 tablet_replica.cc:273] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090: stopping tablet replica I0913 23:14:22.834632 13524 raft_consensus.cc:2147] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 3 FOLLOWER]: Raft consensus shutting down. I0913 23:14:22.834868 13524 raft_consensus.cc:2174] T bf89c16ce260470a863711ddbbdb3d68 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 3 FOLLOWER]: Raft consensus is shut down! I0913 23:14:22.836737 13524 tablet_replica.cc:273] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090: stopping tablet replica I0913 23:14:22.837059 13524 raft_consensus.cc:2147] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 2 FOLLOWER]: Raft consensus shutting down. I0913 23:14:22.837301 13524 raft_consensus.cc:2174] T 82f65eba7703468ca650e7ca50d39549 P 0cad5c4a245f4b3dbd66bb7259f45090 [term 2 FOLLOWER]: Raft consensus is shut down! I0913 23:14:22.877110 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:14:22.877322 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:14:22.877671 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:14:22.879707 15096 raft_consensus.cc:472] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 93a20b201a904b628926e173bbf23463) I0913 23:14:22.880308 15096 raft_consensus.cc:494] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:14:22.882601 15096 leader_election.cc:296] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 93a20b201a904b628926e173bbf23463 (127.13.53.1:44785), 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231) W0913 23:14:22.885529 14645 leader_election.cc:341] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 93a20b201a904b628926e173bbf23463 (127.13.53.1:44785): Network error: Client connection negotiation failed: client connection to 127.13.53.1:44785: connect: Connection refused (error 111) W0913 23:14:22.887392 14643 leader_election.cc:341] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231): Network error: Client connection negotiation failed: client connection to 127.13.53.2:41231: connect: Connection refused (error 111) I0913 23:14:22.887706 14643 leader_election.cc:310] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a66db810508f4e35bac62f5c6c9f6e9c; no voters: 0cad5c4a245f4b3dbd66bb7259f45090, 93a20b201a904b628926e173bbf23463 I0913 23:14:22.888437 15096 raft_consensus.cc:2592] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority I0913 23:14:22.891309 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:14:22.926328 13524 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0913 23:14:22.926635 13524 tablet_replica.cc:273] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c: stopping tablet replica I0913 23:14:22.927117 13524 raft_consensus.cc:2147] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 3 FOLLOWER]: Raft consensus shutting down. I0913 23:14:22.927459 13524 raft_consensus.cc:2174] T 8ad9594f28c64b8ba3dacd888b84312b P a66db810508f4e35bac62f5c6c9f6e9c [term 3 FOLLOWER]: Raft consensus is shut down! I0913 23:14:22.929374 15096 raft_consensus.cc:472] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 93a20b201a904b628926e173bbf23463) I0913 23:14:22.929791 13524 tablet_replica.cc:273] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c: stopping tablet replica I0913 23:14:22.929747 15096 raft_consensus.cc:494] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a20b201a904b628926e173bbf23463" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 44785 } } peers { permanent_uuid: "0cad5c4a245f4b3dbd66bb7259f45090" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 41231 } } peers { permanent_uuid: "a66db810508f4e35bac62f5c6c9f6e9c" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 37091 } } I0913 23:14:22.930238 13524 raft_consensus.cc:2147] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [term 3 FOLLOWER]: Raft consensus shutting down. I0913 23:14:22.930618 13524 raft_consensus.cc:2174] T bf89c16ce260470a863711ddbbdb3d68 P a66db810508f4e35bac62f5c6c9f6e9c [term 3 FOLLOWER]: Raft consensus is shut down! I0913 23:14:22.931201 15096 leader_election.cc:296] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 93a20b201a904b628926e173bbf23463 (127.13.53.1:44785), 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231) W0913 23:14:22.932364 14645 leader_election.cc:341] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 93a20b201a904b628926e173bbf23463 (127.13.53.1:44785): Network error: Client connection negotiation failed: client connection to 127.13.53.1:44785: connect: Connection refused (error 111) W0913 23:14:22.932765 14643 leader_election.cc:341] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0cad5c4a245f4b3dbd66bb7259f45090 (127.13.53.2:41231): Network error: Client connection negotiation failed: client connection to 127.13.53.2:41231: connect: Connection refused (error 111) I0913 23:14:22.933075 14643 leader_election.cc:310] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a66db810508f4e35bac62f5c6c9f6e9c; no voters: 0cad5c4a245f4b3dbd66bb7259f45090, 93a20b201a904b628926e173bbf23463 I0913 23:14:22.933457 13524 tablet_replica.cc:273] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c: stopping tablet replica I0913 23:14:22.933609 15096 raft_consensus.cc:2592] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0913 23:14:22.933959 13524 raft_consensus.cc:2147] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [term 2 FOLLOWER]: Raft consensus shutting down. I0913 23:14:22.934340 13524 raft_consensus.cc:2174] T 82f65eba7703468ca650e7ca50d39549 P a66db810508f4e35bac62f5c6c9f6e9c [term 2 FOLLOWER]: Raft consensus is shut down! I0913 23:14:22.956066 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:14:22.956251 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:14:22.956559 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:14:22.968951 13524 master.cc:247] Master@127.13.53.62:40439 shutting down... I0913 23:14:22.992161 13524 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 6 FOLLOWER]: Raft consensus shutting down. I0913 23:14:22.992527 13524 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b [term 6 FOLLOWER]: Raft consensus is shut down! I0913 23:14:22.992760 13524 tablet_replica.cc:273] T 00000000000000000000000000000000 P 46574f27d20e427381df2ea44498c90b: stopping tablet replica I0913 23:14:23.022583 13524 master.cc:258] Master@127.13.53.62:40439 shutdown complete. I0913 23:14:23.031311 13524 master.cc:247] Master@127.13.53.61:35977 shutting down... I0913 23:14:23.053100 13524 raft_consensus.cc:2147] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 6 LEADER]: Raft consensus shutting down. I0913 23:14:23.053864 13524 raft_consensus.cc:2174] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca [term 6 FOLLOWER]: Raft consensus is shut down! I0913 23:14:23.054172 13524 tablet_replica.cc:273] T 00000000000000000000000000000000 P b03280a8df7e48f9939e076d6d0138ca: stopping tablet replica I0913 23:14:23.071416 13524 master.cc:258] Master@127.13.53.61:35977 shutdown complete. I0913 23:14:23.081518 13524 master.cc:247] Master@127.13.53.60:42667 shutting down... I0913 23:14:23.103221 13524 raft_consensus.cc:2147] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 6 FOLLOWER]: Raft consensus shutting down. I0913 23:14:23.103585 13524 raft_consensus.cc:2174] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607 [term 6 FOLLOWER]: Raft consensus is shut down! I0913 23:14:23.103798 13524 tablet_replica.cc:273] T 00000000000000000000000000000000 P fab3d6e5bcad48acbb0fe6b928638607: stopping tablet replica I0913 23:14:23.120494 13524 master.cc:258] Master@127.13.53.60:42667 shutdown complete. I0913 23:14:23.133746 13524 test_util.cc:135] ----------------------------------------------- I0913 23:14:23.133925 13524 test_util.cc:136] Had fatal failures, leaving test files at /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568416417781642-13524 [ FAILED ] TsTabletManagerITest.TestTableStats (36307 ms) [----------] 4 tests from TsTabletManagerITest (45274 ms total) [----------] 2 tests from FailedTabletsAreReplacedITest [ RUN ] FailedTabletsAreReplacedITest.OneReplica/0 I0913 23:14:23.140087 13524 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.13.53.62:44755 I0913 23:14:23.151324 13524 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 36700us I0913 23:14:23.151743 13524 fs_manager.cc:263] Metadata directory not provided I0913 23:14:23.151870 13524 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0913 23:14:23.152097 13524 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0913 23:14:23.152201 13524 server_base.cc:449] Attempting to create new FS layout instead I0913 23:14:23.153254 13524 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568416417781642-13524/minicluster-data/master-0-root/instance: uuid: "d700d56f713b46b48b7b5d90e96765db" format_stamp: "Formatted at 2019-09-13 23:14:23 on dist-test-slave-dist-test-slave-kz3c" I0913 23:14:23.161239 13524 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.007s sys 0.003s I0913 23:14:23.162746 13524 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0913 23:14:23.168841 13524 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0913 23:14:23.169003 13524 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568416417781642-13524/minicluster-data/master-0-root uuid: "d700d56f713b46b48b7b5d90e96765db" format_stamp: "Formatted at 2019-09-13 23:14:23 on dist-test-slave-dist-test-slave-kz3c" I0913 23:14:23.169246 13524 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568416417781642-13524/minicluster-data/master-0-root metadata directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568416417781642-13524/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568416417781642-13524/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0913 23:14:23.231979 13524 rpc_server.cc:205] RPC server started. Bound to: 127.13.53.62:44755 I0913 23:14:23.233850 13524 webserver.cc:374] Webserver started at http://127.13.53.62:46325/ using document root and password file I0913 23:14:23.236057 15157 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0913 23:14:23.244681 15157 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P d700d56f713b46b48b7b5d90e96765db: Bootstrap starting. I0913 23:14:23.247402 15157 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P d700d56f713b46b48b7b5d90e96765db: No blocks or log segments found. Creating new log. I0913 23:14:23.250754 15157 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P d700d56f713b46b48b7b5d90e96765db: No bootstrap required, opened a new log I0913 23:14:23.252243 15157 raft_consensus.cc:340] T 00000000000000000000000000000000 P d700d56f713b46b48b7b5d90e96765db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d700d56f713b46b48b7b5d90e96765db" member_type: VOTER } I0913 23:14:23.252709 15157 raft_consensus.cc:366] T 00000000000000000000000000000000 P d700d56f713b46b48b7b5d90e96765db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0913 23:14:23.252876 15157 raft_consensus.cc:705] T 00000000000000000000000000000000 P d700d56f713b46b48b7b5d90e96765db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d700d56f713b46b48b7b5d90e96765db, State: Initialized, Role: FOLLOWER I0913 23:14:23.253307 15157 consensus_queue.cc:229] T 00000000000000000000000000000000 P d700d56f713b46b48b7b5d90e96765db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d700d56f713b46b48b7b5d90e96765db" member_type: VOTER } I0913 23:14:23.253705 15157 raft_consensus.cc:380] T 00000000000000000000000000000000 P d700d56f713b46b48b7b5d90e96765db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0913 23:14:23.253841 15157 raft_consensus.cc:472] T 00000000000000000000000000000000 P d700d56f713b46b48b7b5d90e96765db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0913 23:14:23.254020 15157 raft_consensus.cc:2890] T 00000000000000000000000000000000 P d700d56f713b46b48b7b5d90e96765db [term 0 FOLLOWER]: Advancing to term 1 I0913 23:14:23.257757 15157 raft_consensus.cc:494] T 00000000000000000000000000000000 P d700d56f713b46b48b7b5d90e96765db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d700d56f713b46b48b7b5d90e96765db" member_type: VOTER } I0913 23:14:23.258252 15157 leader_election.cc:310] T 00000000000000000000000000000000 P d700d56f713b46b48b7b5d90e96765db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d700d56f713b46b48b7b5d90e96765db; no voters: I0913 23:14:23.259436 15157 leader_election.cc:296] T 00000000000000000000000000000000 P d700d56f713b46b48b7b5d90e96765db [CANDIDATE]: Term 1 election: Requested vote from peers I0913 23:14:23.259814 15159 raft_consensus.cc:2644] T 00000000000000000000000000000000 P d700d56f713b46b48b7b5d90e96765db [term 1 FOLLOWER]: Leader election won for term 1 I0913 23:14:23.261173 15159 raft_consensus.cc:667] T 00000000000000000000000000000000 P d700d56f713b46b48b7b5d90e96765db [term 1 LEADER]: Becoming Leader. State: Replica: d700d56f713b46b48b7b5d90e96765db, State: Running, Role: LEADER I0913 23:14:23.261569 15159 consensus_queue.cc:206] T 00000000000000000000000000000000 P d700d56f713b46b48b7b5d90e96765db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d700d56f713b46b48b7b5d90e96765db" member_type: VOTER } I0913 23:14:23.264583 15159 sys_catalog.cc:340] T 00000000000000000000000000000000 P d700d56f713b46b48b7b5d90e96765db [sys.catalog]: SysCatalogTable state changed. Reason: New leader d700d56f713b46b48b7b5d90e96765db. Latest consensus state: current_term: 1 leader_uuid: "d700d56f713b46b48b7b5d90e96765db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d700d56f713b46b48b7b5d90e96765db" member_type: VOTER } } I0913 23:14:23.264952 15159 sys_catalog.cc:343] T 00000000000000000000000000000000 P d700d56f713b46b48b7b5d90e96765db [sys.catalog]: This master's current role is: LEADER I0913 23:14:23.265709 15161 sys_catalog.cc:340] T 00000000000000000000000000000000 P d700d56f713b46b48b7b5d90e96765db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d700d56f713b46b48b7b5d90e96765db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d700d56f713b46b48b7b5d90e96765db" member_type: VOTER } } I0913 23:14:23.266480 15161 sys_catalog.cc:343] T 00000000000000000000000000000000 P d700d56f713b46b48b7b5d90e96765db [sys.catalog]: This master's current role is: LEADER I0913 23:14:23.269064 15157 sys_catalog.cc:340] T 00000000000000000000000000000000 P d700d56f713b46b48b7b5d90e96765db [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "d700d56f713b46b48b7b5d90e96765db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d700d56f713b46b48b7b5d90e96765db" member_type: VOTER } } I0913 23:14:23.269402 15157 sys_catalog.cc:343] T 00000000000000000000000000000000 P d700d56f713b46b48b7b5d90e96765db [sys.catalog]: This master's current role is: LEADER I0913 23:14:23.270305 15157 sys_catalog.cc:428] T 00000000000000000000000000000000 P d700d56f713b46b48b7b5d90e96765db [sys.catalog]: configured and running, proceeding with master startup. I0913 23:14:23.271083 15164 catalog_manager.cc:1071] Loading table and tablet metadata into memory... I0913 23:14:23.271533 15166 catalog_manager.cc:554] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting I0913 23:14:23.273658 15164 catalog_manager.cc:1082] Initializing Kudu internal certificate authority... I0913 23:14:23.278296 13524 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0913 23:14:23.301630 15164 catalog_manager.cc:966] Generated new certificate authority record I0913 23:14:23.303261 15164 catalog_manager.cc:1091] Loading token signing keys... I0913 23:14:23.314684 15164 catalog_manager.cc:4394] T 00000000000000000000000000000000 P d700d56f713b46b48b7b5d90e96765db: Generated new TSK 0 I0913 23:14:23.355304 13524 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 36700us I0913 23:14:23.355687 13524 fs_manager.cc:263] Metadata directory not provided I0913 23:14:23.355834 13524 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0913 23:14:23.356045 13524 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0913 23:14:23.356149 13524 server_base.cc:449] Attempting to create new FS layout instead I0913 23:14:23.357125 13524 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568416417781642-13524/minicluster-data/ts-0-root/instance: uuid: "63ebaa75f74e4547af25dfa41053b1f6" format_stamp: "Formatted at 2019-09-13 23:14:23 on dist-test-slave-dist-test-slave-kz3c" I0913 23:14:23.365373 13524 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.010s sys 0.001s I0913 23:14:23.366910 13524 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0913 23:14:23.373093 13524 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0913 23:14:23.373257 13524 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568416417781642-13524/minicluster-data/ts-0-root uuid: "63ebaa75f74e4547af25dfa41053b1f6" format_stamp: "Formatted at 2019-09-13 23:14:23 on dist-test-slave-dist-test-slave-kz3c" I0913 23:14:23.373498 13524 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568416417781642-13524/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568416417781642-13524/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568416417781642-13524/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0913 23:14:23.439126 13524 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0913 23:14:23.439285 13524 ts_tablet_manager.cc:403] Registered 0 tablets I0913 23:14:23.477705 13524 rpc_server.cc:205] RPC server started. Bound to: 127.13.53.1:40379 I0913 23:14:23.479542 13524 webserver.cc:374] Webserver started at http://127.13.53.1:41411/ using document root and password file I0913 23:14:23.494783 13524 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 36700us I0913 23:14:23.495136 13524 fs_manager.cc:263] Metadata directory not provided I0913 23:14:23.495278 13524 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0913 23:14:23.495455 13524 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0913 23:14:23.495560 13524 server_base.cc:449] Attempting to create new FS layout instead I0913 23:14:23.496541 13524 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568416417781642-13524/minicluster-data/ts-1-root/instance: uuid: "65475b138e8547789c377ceee9656e60" format_stamp: "Formatted at 2019-09-13 23:14:23 on dist-test-slave-dist-test-slave-kz3c" I0913 23:14:23.499037 15232 heartbeater.cc:322] Connected to a master server at 127.13.53.62:44755 I0913 23:14:23.499277 15232 heartbeater.cc:406] Registering TS with master... I0913 23:14:23.499672 15232 heartbeater.cc:455] Master 127.13.53.62:44755 requested a full tablet report, sending... I0913 23:14:23.501188 15124 ts_manager.cc:161] Registered new tserver with Master: 63ebaa75f74e4547af25dfa41053b1f6 (127.13.53.1:40379) I0913 23:14:23.503000 15124 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59382 I0913 23:14:23.506192 13524 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.011s sys 0.001s I0913 23:14:23.507647 13524 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0913 23:14:23.514958 13524 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0913 23:14:23.515164 13524 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568416417781642-13524/minicluster-data/ts-1-root uuid: "65475b138e8547789c377ceee9656e60" format_stamp: "Formatted at 2019-09-13 23:14:23 on dist-test-slave-dist-test-slave-kz3c" I0913 23:14:23.515395 13524 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568416417781642-13524/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568416417781642-13524/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568416417781642-13524/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0913 23:14:23.540030 13524 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0913 23:14:23.540221 13524 ts_tablet_manager.cc:403] Registered 0 tablets I0913 23:14:23.581182 13524 rpc_server.cc:205] RPC server started. Bound to: 127.13.53.2:32899 I0913 23:14:23.583010 13524 webserver.cc:374] Webserver started at http://127.13.53.2:41741/ using document root and password file I0913 23:14:23.596699 15300 heartbeater.cc:322] Connected to a master server at 127.13.53.62:44755 I0913 23:14:23.597056 15300 heartbeater.cc:406] Registering TS with master... I0913 23:14:23.597676 15300 heartbeater.cc:455] Master 127.13.53.62:44755 requested a full tablet report, sending... I0913 23:14:23.599298 15124 ts_manager.cc:161] Registered new tserver with Master: 65475b138e8547789c377ceee9656e60 (127.13.53.2:32899) I0913 23:14:23.600596 15124 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59384 I0913 23:14:23.602239 13524 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 36700us I0913 23:14:23.602645 13524 fs_manager.cc:263] Metadata directory not provided I0913 23:14:23.602818 13524 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0913 23:14:23.603046 13524 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0913 23:14:23.603185 13524 server_base.cc:449] Attempting to create new FS layout instead I0913 23:14:23.604338 13524 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568416417781642-13524/minicluster-data/ts-2-root/instance: uuid: "afe09d31e65f437faa6e61008d02c307" format_stamp: "Formatted at 2019-09-13 23:14:23 on dist-test-slave-dist-test-slave-kz3c" I0913 23:14:23.614697 13524 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.012s sys 0.000s I0913 23:14:23.616269 13524 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0913 23:14:23.623627 13524 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0913 23:14:23.623826 13524 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568416417781642-13524/minicluster-data/ts-2-root uuid: "afe09d31e65f437faa6e61008d02c307" format_stamp: "Formatted at 2019-09-13 23:14:23 on dist-test-slave-dist-test-slave-kz3c" I0913 23:14:23.624078 13524 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568416417781642-13524/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568416417781642-13524/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568416417781642-13524/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0913 23:14:23.665544 13524 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0913 23:14:23.665736 13524 ts_tablet_manager.cc:403] Registered 0 tablets I0913 23:14:23.707801 13524 rpc_server.cc:205] RPC server started. Bound to: 127.13.53.3:44889 I0913 23:14:23.709581 13524 webserver.cc:374] Webserver started at http://127.13.53.3:37127/ using document root and password file I0913 23:14:23.723840 15367 heartbeater.cc:322] Connected to a master server at 127.13.53.62:44755 I0913 23:14:23.724195 15367 heartbeater.cc:406] Registering TS with master... I0913 23:14:23.724648 15367 heartbeater.cc:455] Master 127.13.53.62:44755 requested a full tablet report, sending... I0913 23:14:23.726337 15124 ts_manager.cc:161] Registered new tserver with Master: afe09d31e65f437faa6e61008d02c307 (127.13.53.3:44889) I0913 23:14:23.727113 13524 internal_mini_cluster.cc:328] 3 TS(s) registered with all masters after 0.015475963s I0913 23:14:23.729281 15124 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59386 I0913 23:14:23.746533 13524 test_util.cc:212] Using random seed: 1781429706 I0913 23:14:23.767357 15124 catalog_manager.cc:1425] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59390: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0913 23:14:23.801873 15267 tablet_service.cc:1022] Processing CreateTablet for tablet f9b2cffa071640fbb75fb8b0430cb6f7 (table=test-workload [id=93f0414fe23945a1b6acf86c397866f3]), partition=RANGE (key) PARTITION UNBOUNDED I0913 23:14:23.809100 15267 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet f9b2cffa071640fbb75fb8b0430cb6f7. 1 dirs total, 0 dirs full, 0 dirs failed I0913 23:14:23.822258 15386 tablet_bootstrap.cc:439] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60: Bootstrap starting. I0913 23:14:23.825481 15199 tablet_service.cc:1022] Processing CreateTablet for tablet f9b2cffa071640fbb75fb8b0430cb6f7 (table=test-workload [id=93f0414fe23945a1b6acf86c397866f3]), partition=RANGE (key) PARTITION UNBOUNDED I0913 23:14:23.825686 15334 tablet_service.cc:1022] Processing CreateTablet for tablet f9b2cffa071640fbb75fb8b0430cb6f7 (table=test-workload [id=93f0414fe23945a1b6acf86c397866f3]), partition=RANGE (key) PARTITION UNBOUNDED I0913 23:14:23.825772 15386 tablet_bootstrap.cc:590] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60: No blocks or log segments found. Creating new log. I0913 23:14:23.826674 15199 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet f9b2cffa071640fbb75fb8b0430cb6f7. 1 dirs total, 0 dirs full, 0 dirs failed I0913 23:14:23.826684 15334 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet f9b2cffa071640fbb75fb8b0430cb6f7. 1 dirs total, 0 dirs full, 0 dirs failed I0913 23:14:23.837003 15386 tablet_bootstrap.cc:439] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60: No bootstrap required, opened a new log I0913 23:14:23.837683 15386 ts_tablet_manager.cc:1085] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.010s I0913 23:14:23.839965 15386 raft_consensus.cc:340] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65475b138e8547789c377ceee9656e60" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 32899 } } peers { permanent_uuid: "63ebaa75f74e4547af25dfa41053b1f6" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 40379 } } peers { permanent_uuid: "afe09d31e65f437faa6e61008d02c307" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 44889 } } I0913 23:14:23.840848 15386 raft_consensus.cc:366] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0913 23:14:23.841157 15386 raft_consensus.cc:705] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65475b138e8547789c377ceee9656e60, State: Initialized, Role: FOLLOWER I0913 23:14:23.842221 15388 tablet_bootstrap.cc:439] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6: Bootstrap starting. I0913 23:14:23.842113 15386 consensus_queue.cc:229] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65475b138e8547789c377ceee9656e60" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 32899 } } peers { permanent_uuid: "63ebaa75f74e4547af25dfa41053b1f6" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 40379 } } peers { permanent_uuid: "afe09d31e65f437faa6e61008d02c307" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 44889 } } I0913 23:14:23.844362 15389 tablet_bootstrap.cc:439] T f9b2cffa071640fbb75fb8b0430cb6f7 P afe09d31e65f437faa6e61008d02c307: Bootstrap starting. I0913 23:14:23.845762 15300 heartbeater.cc:447] Master 127.13.53.62:44755 was elected leader, sending a full tablet report... I0913 23:14:23.845902 15386 ts_tablet_manager.cc:1113] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s I0913 23:14:23.848229 15388 tablet_bootstrap.cc:590] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6: No blocks or log segments found. Creating new log. I0913 23:14:23.848770 15389 tablet_bootstrap.cc:590] T f9b2cffa071640fbb75fb8b0430cb6f7 P afe09d31e65f437faa6e61008d02c307: No blocks or log segments found. Creating new log. I0913 23:14:23.852546 15389 tablet_bootstrap.cc:439] T f9b2cffa071640fbb75fb8b0430cb6f7 P afe09d31e65f437faa6e61008d02c307: No bootstrap required, opened a new log I0913 23:14:23.852910 15389 ts_tablet_manager.cc:1085] T f9b2cffa071640fbb75fb8b0430cb6f7 P afe09d31e65f437faa6e61008d02c307: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s I0913 23:14:23.853190 15388 tablet_bootstrap.cc:439] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6: No bootstrap required, opened a new log I0913 23:14:23.853633 15388 ts_tablet_manager.cc:1085] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.003s I0913 23:14:23.854444 15389 raft_consensus.cc:340] T f9b2cffa071640fbb75fb8b0430cb6f7 P afe09d31e65f437faa6e61008d02c307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65475b138e8547789c377ceee9656e60" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 32899 } } peers { permanent_uuid: "63ebaa75f74e4547af25dfa41053b1f6" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 40379 } } peers { permanent_uuid: "afe09d31e65f437faa6e61008d02c307" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 44889 } } I0913 23:14:23.854977 15389 raft_consensus.cc:366] T f9b2cffa071640fbb75fb8b0430cb6f7 P afe09d31e65f437faa6e61008d02c307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0913 23:14:23.855129 15389 raft_consensus.cc:705] T f9b2cffa071640fbb75fb8b0430cb6f7 P afe09d31e65f437faa6e61008d02c307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afe09d31e65f437faa6e61008d02c307, State: Initialized, Role: FOLLOWER I0913 23:14:23.855221 15388 raft_consensus.cc:340] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65475b138e8547789c377ceee9656e60" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 32899 } } peers { permanent_uuid: "63ebaa75f74e4547af25dfa41053b1f6" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 40379 } } peers { permanent_uuid: "afe09d31e65f437faa6e61008d02c307" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 44889 } } I0913 23:14:23.855832 15388 raft_consensus.cc:366] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0913 23:14:23.855556 15389 consensus_queue.cc:229] T f9b2cffa071640fbb75fb8b0430cb6f7 P afe09d31e65f437faa6e61008d02c307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65475b138e8547789c377ceee9656e60" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 32899 } } peers { permanent_uuid: "63ebaa75f74e4547af25dfa41053b1f6" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 40379 } } peers { permanent_uuid: "afe09d31e65f437faa6e61008d02c307" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 44889 } } I0913 23:14:23.856016 15388 raft_consensus.cc:705] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ebaa75f74e4547af25dfa41053b1f6, State: Initialized, Role: FOLLOWER I0913 23:14:23.856501 15388 consensus_queue.cc:229] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65475b138e8547789c377ceee9656e60" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 32899 } } peers { permanent_uuid: "63ebaa75f74e4547af25dfa41053b1f6" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 40379 } } peers { permanent_uuid: "afe09d31e65f437faa6e61008d02c307" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 44889 } } I0913 23:14:23.857314 15367 heartbeater.cc:447] Master 127.13.53.62:44755 was elected leader, sending a full tablet report... I0913 23:14:23.857506 15389 ts_tablet_manager.cc:1113] T f9b2cffa071640fbb75fb8b0430cb6f7 P afe09d31e65f437faa6e61008d02c307: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s I0913 23:14:23.858642 15232 heartbeater.cc:447] Master 127.13.53.62:44755 was elected leader, sending a full tablet report... I0913 23:14:23.859808 15388 ts_tablet_manager.cc:1113] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s W0913 23:14:23.962340 15368 tablet.cc:1762] T f9b2cffa071640fbb75fb8b0430cb6f7 P afe09d31e65f437faa6e61008d02c307: Can't schedule compaction. Clean time has not been advanced past its initial value. I0913 23:14:23.993470 15390 raft_consensus.cc:472] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0913 23:14:23.993947 15390 raft_consensus.cc:494] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65475b138e8547789c377ceee9656e60" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 32899 } } peers { permanent_uuid: "63ebaa75f74e4547af25dfa41053b1f6" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 40379 } } peers { permanent_uuid: "afe09d31e65f437faa6e61008d02c307" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 44889 } } I0913 23:14:23.995405 15390 leader_election.cc:296] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63ebaa75f74e4547af25dfa41053b1f6 (127.13.53.1:40379), afe09d31e65f437faa6e61008d02c307 (127.13.53.3:44889) I0913 23:14:24.001633 15393 raft_consensus.cc:472] T f9b2cffa071640fbb75fb8b0430cb6f7 P afe09d31e65f437faa6e61008d02c307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0913 23:14:24.002203 15393 raft_consensus.cc:494] T f9b2cffa071640fbb75fb8b0430cb6f7 P afe09d31e65f437faa6e61008d02c307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65475b138e8547789c377ceee9656e60" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 32899 } } peers { permanent_uuid: "63ebaa75f74e4547af25dfa41053b1f6" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 40379 } } peers { permanent_uuid: "afe09d31e65f437faa6e61008d02c307" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 44889 } } I0913 23:14:24.009347 15393 leader_election.cc:296] T f9b2cffa071640fbb75fb8b0430cb6f7 P afe09d31e65f437faa6e61008d02c307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65475b138e8547789c377ceee9656e60 (127.13.53.2:32899), 63ebaa75f74e4547af25dfa41053b1f6 (127.13.53.1:40379) I0913 23:14:24.012662 15209 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f9b2cffa071640fbb75fb8b0430cb6f7" candidate_uuid: "65475b138e8547789c377ceee9656e60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ebaa75f74e4547af25dfa41053b1f6" is_pre_election: true I0913 23:14:24.013572 15209 raft_consensus.cc:2350] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65475b138e8547789c377ceee9656e60 in term 0. I0913 23:14:24.015280 15239 leader_election.cc:310] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63ebaa75f74e4547af25dfa41053b1f6, 65475b138e8547789c377ceee9656e60; no voters: I0913 23:14:24.016113 15390 raft_consensus.cc:2644] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0913 23:14:24.016387 15390 raft_consensus.cc:472] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0913 23:14:24.016760 15390 raft_consensus.cc:2890] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 [term 0 FOLLOWER]: Advancing to term 1 I0913 23:14:24.017122 15344 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f9b2cffa071640fbb75fb8b0430cb6f7" candidate_uuid: "65475b138e8547789c377ceee9656e60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afe09d31e65f437faa6e61008d02c307" is_pre_election: true I0913 23:14:24.017678 15344 raft_consensus.cc:2350] T f9b2cffa071640fbb75fb8b0430cb6f7 P afe09d31e65f437faa6e61008d02c307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65475b138e8547789c377ceee9656e60 in term 0. I0913 23:14:24.019304 15277 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f9b2cffa071640fbb75fb8b0430cb6f7" candidate_uuid: "afe09d31e65f437faa6e61008d02c307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65475b138e8547789c377ceee9656e60" is_pre_election: true I0913 23:14:24.021518 15390 raft_consensus.cc:494] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65475b138e8547789c377ceee9656e60" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 32899 } } peers { permanent_uuid: "63ebaa75f74e4547af25dfa41053b1f6" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 40379 } } peers { permanent_uuid: "afe09d31e65f437faa6e61008d02c307" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 44889 } } I0913 23:14:24.022544 15277 raft_consensus.cc:2280] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate afe09d31e65f437faa6e61008d02c307 in current term 1: Already voted for candidate 65475b138e8547789c377ceee9656e60 in this term. I0913 23:14:24.022943 15390 leader_election.cc:296] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 [CANDIDATE]: Term 1 election: Requested vote from peers 63ebaa75f74e4547af25dfa41053b1f6 (127.13.53.1:40379), afe09d31e65f437faa6e61008d02c307 (127.13.53.3:44889) I0913 23:14:24.023970 15344 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f9b2cffa071640fbb75fb8b0430cb6f7" candidate_uuid: "65475b138e8547789c377ceee9656e60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afe09d31e65f437faa6e61008d02c307" I0913 23:14:24.024608 15344 raft_consensus.cc:2890] T f9b2cffa071640fbb75fb8b0430cb6f7 P afe09d31e65f437faa6e61008d02c307 [term 0 FOLLOWER]: Advancing to term 1 I0913 23:14:24.023926 15209 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f9b2cffa071640fbb75fb8b0430cb6f7" candidate_uuid: "65475b138e8547789c377ceee9656e60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ebaa75f74e4547af25dfa41053b1f6" I0913 23:14:24.026284 15209 raft_consensus.cc:2890] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6 [term 0 FOLLOWER]: Advancing to term 1 I0913 23:14:24.031420 15208 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f9b2cffa071640fbb75fb8b0430cb6f7" candidate_uuid: "afe09d31e65f437faa6e61008d02c307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63ebaa75f74e4547af25dfa41053b1f6" is_pre_election: true I0913 23:14:24.033236 15344 raft_consensus.cc:2350] T f9b2cffa071640fbb75fb8b0430cb6f7 P afe09d31e65f437faa6e61008d02c307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65475b138e8547789c377ceee9656e60 in term 1. I0913 23:14:24.034013 15209 raft_consensus.cc:2350] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65475b138e8547789c377ceee9656e60 in term 1. I0913 23:14:24.034258 15239 leader_election.cc:310] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65475b138e8547789c377ceee9656e60, afe09d31e65f437faa6e61008d02c307; no voters: I0913 23:14:24.034570 15208 raft_consensus.cc:2325] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate afe09d31e65f437faa6e61008d02c307 for term 1 because replica is already servicing an update from a current leader or another vote. I0913 23:14:24.035117 15390 raft_consensus.cc:2644] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 [term 1 FOLLOWER]: Leader election won for term 1 I0913 23:14:24.036078 15306 leader_election.cc:310] T f9b2cffa071640fbb75fb8b0430cb6f7 P afe09d31e65f437faa6e61008d02c307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: afe09d31e65f437faa6e61008d02c307; no voters: 63ebaa75f74e4547af25dfa41053b1f6, 65475b138e8547789c377ceee9656e60 I0913 23:14:24.036392 15390 raft_consensus.cc:667] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 [term 1 LEADER]: Becoming Leader. State: Replica: 65475b138e8547789c377ceee9656e60, State: Running, Role: LEADER I0913 23:14:24.036945 15393 raft_consensus.cc:2592] T f9b2cffa071640fbb75fb8b0430cb6f7 P afe09d31e65f437faa6e61008d02c307 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0913 23:14:24.037112 15390 consensus_queue.cc:206] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65475b138e8547789c377ceee9656e60" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 32899 } } peers { permanent_uuid: "63ebaa75f74e4547af25dfa41053b1f6" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 40379 } } peers { permanent_uuid: "afe09d31e65f437faa6e61008d02c307" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 44889 } } I0913 23:14:24.042593 15124 catalog_manager.cc:4079] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 reported cstate change: term changed from 0 to 1, leader changed from to 65475b138e8547789c377ceee9656e60 (127.13.53.2). New cstate: current_term: 1 leader_uuid: "65475b138e8547789c377ceee9656e60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65475b138e8547789c377ceee9656e60" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 32899 } } peers { permanent_uuid: "63ebaa75f74e4547af25dfa41053b1f6" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 40379 } } peers { permanent_uuid: "afe09d31e65f437faa6e61008d02c307" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 44889 } } } Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- d700d56f713b46b48b7b5d90e96765db | 127.13.53.62:44755 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+-------------------+---------+---------- 63ebaa75f74e4547af25dfa41053b1f6 | 127.13.53.1:40379 | HEALTHY | 65475b138e8547789c377ceee9656e60 | 127.13.53.2:32899 | HEALTHY | afe09d31e65f437faa6e61008d02c307 | 127.13.53.3:44889 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 4 server(s) checked Tablet Summary Tablet f9b2cffa071640fbb75fb8b0430cb6f7 of table 'test-workload' is conflicted: 3 replicas' active configs disagree with the leader master's 65475b138e8547789c377ceee9656e60 (127.13.53.2:32899): RUNNING [LEADER] 63ebaa75f74e4547af25dfa41053b1f6 (127.13.53.1:40379): RUNNING afe09d31e65f437faa6e61008d02c307 (127.13.53.3:44889): RUNNING All reported replicas are: A = 65475b138e8547789c377ceee9656e60 B = 63ebaa75f74e4547af25dfa41053b1f6 C = afe09d31e65f437faa6e61008d02c307 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 1 | -1 | Yes B | A B C | 1 | -1 | Yes C | A B C | 1 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0913 23:14:24.229396 13524 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- d700d56f713b46b48b7b5d90e96765db | 127.13.53.62:44755 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+-------------------+---------+---------- 63ebaa75f74e4547af25dfa41053b1f6 | 127.13.53.1:40379 | HEALTHY | 65475b138e8547789c377ceee9656e60 | 127.13.53.2:32899 | HEALTHY | afe09d31e65f437faa6e61008d02c307 | 127.13.53.3:44889 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 4 server(s) checked Tablet Summary Tablet f9b2cffa071640fbb75fb8b0430cb6f7 of table 'test-workload' is conflicted: 3 replicas' active configs disagree with the leader master's 65475b138e8547789c377ceee9656e60 (127.13.53.2:32899): RUNNING [LEADER] 63ebaa75f74e4547af25dfa41053b1f6 (127.13.53.1:40379): RUNNING afe09d31e65f437faa6e61008d02c307 (127.13.53.3:44889): RUNNING All reported replicas are: A = 65475b138e8547789c377ceee9656e60 B = 63ebaa75f74e4547af25dfa41053b1f6 C = afe09d31e65f437faa6e61008d02c307 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 1 | -1 | Yes B | A B C | 1 | -1 | Yes C | A B C | 1 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0913 23:14:24.541497 13524 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0913 23:14:24.574452 15448 consensus_queue.cc:984] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ebaa75f74e4547af25dfa41053b1f6" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 40379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0913 23:14:24.584856 15448 consensus_queue.cc:984] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afe09d31e65f437faa6e61008d02c307" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 44889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- d700d56f713b46b48b7b5d90e96765db | 127.13.53.62:44755 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+-------------------+---------+---------- 63ebaa75f74e4547af25dfa41053b1f6 | 127.13.53.1:40379 | HEALTHY | 65475b138e8547789c377ceee9656e60 | 127.13.53.2:32899 | HEALTHY | afe09d31e65f437faa6e61008d02c307 | 127.13.53.3:44889 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 4 server(s) checked Tablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+---------+---------------+---------+------------+------------------+------------- test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0913 23:14:24.927737 13524 log_verifier.cc:121] Checking tablet f9b2cffa071640fbb75fb8b0430cb6f7 I0913 23:14:24.994834 13524 log_verifier.cc:172] Verified matching terms for 1 ops in tablet f9b2cffa071640fbb75fb8b0430cb6f7 I0913 23:14:24.995297 13524 tablet_replica.cc:273] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6: stopping tablet replica I0913 23:14:24.995645 13524 raft_consensus.cc:2147] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6 [term 1 FOLLOWER]: Raft consensus shutting down. I0913 23:14:24.995910 13524 raft_consensus.cc:2174] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6 [term 1 FOLLOWER]: Raft consensus is shut down! W0913 23:14:25.122964 15239 consensus_peers.cc:458] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 -> Peer 63ebaa75f74e4547af25dfa41053b1f6 (127.13.53.1:40379): Couldn't send request to peer 63ebaa75f74e4547af25dfa41053b1f6. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry. I0913 23:14:25.128983 15448 raft_consensus.cc:922] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60: Attempting to remove follower 63ebaa75f74e4547af25dfa41053b1f6 from the Raft config. Reason: The tablet replica hosted on peer 63ebaa75f74e4547af25dfa41053b1f6 has failed I0913 23:14:25.130388 15448 consensus_queue.cc:206] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "65475b138e8547789c377ceee9656e60" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 32899 } } peers { permanent_uuid: "afe09d31e65f437faa6e61008d02c307" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 44889 } } I0913 23:14:25.133755 15344 raft_consensus.cc:1184] T f9b2cffa071640fbb75fb8b0430cb6f7 P afe09d31e65f437faa6e61008d02c307 [term 1 FOLLOWER]: Refusing update from remote peer 65475b138e8547789c377ceee9656e60: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0913 23:14:25.134783 15448 consensus_queue.cc:984] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afe09d31e65f437faa6e61008d02c307" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 44889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0913 23:14:25.138962 15500 raft_consensus.cc:2792] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 63ebaa75f74e4547af25dfa41053b1f6 (127.13.53.1) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "65475b138e8547789c377ceee9656e60" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 32899 } } peers { permanent_uuid: "afe09d31e65f437faa6e61008d02c307" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 44889 } } } I0913 23:14:25.140142 15344 raft_consensus.cc:2792] T f9b2cffa071640fbb75fb8b0430cb6f7 P afe09d31e65f437faa6e61008d02c307 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 63ebaa75f74e4547af25dfa41053b1f6 (127.13.53.1) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "65475b138e8547789c377ceee9656e60" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 32899 } } peers { permanent_uuid: "afe09d31e65f437faa6e61008d02c307" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 44889 } } } Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- d700d56f713b46b48b7b5d90e96765db | 127.13.53.62:44755 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+-------------------+---------+---------- 63ebaa75f74e4547af25dfa41053b1f6 | 127.13.53.1:40379 | HEALTHY | 65475b138e8547789c377ceee9656e60 | 127.13.53.2:32899 | HEALTHY | afe09d31e65f437faa6e61008d02c307 | 127.13.53.3:44889 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 4 server(s) checked Tablet Summary Tablet f9b2cffa071640fbb75fb8b0430cb6f7 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 65475b138e8547789c377ceee9656e60 (127.13.53.2:32899): RUNNING [LEADER] 63ebaa75f74e4547af25dfa41053b1f6 (127.13.53.1:40379): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: INJECTED ERROR: tablet failed afe09d31e65f437faa6e61008d02c307 (127.13.53.3:44889): RUNNING All reported replicas are: A = 65475b138e8547789c377ceee9656e60 B = 63ebaa75f74e4547af25dfa41053b1f6 C = afe09d31e65f437faa6e61008d02c307 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 1 | -1 | Yes B | [config not available] | | | C | A* B C | 1 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0913 23:14:25.151302 13524 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0913 23:14:25.149158 15122 catalog_manager.cc:4079] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 reported cstate change: config changed from index -1 to 2, VOTER 63ebaa75f74e4547af25dfa41053b1f6 (127.13.53.1) evicted. New cstate: current_term: 1 leader_uuid: "65475b138e8547789c377ceee9656e60" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "65475b138e8547789c377ceee9656e60" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 32899 } } peers { permanent_uuid: "afe09d31e65f437faa6e61008d02c307" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 44889 } } } I0913 23:14:25.163466 15122 catalog_manager.cc:3471] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet f9b2cffa071640fbb75fb8b0430cb6f7 on 63ebaa75f74e4547af25dfa41053b1f6 (127.13.53.1:40379) (TS 63ebaa75f74e4547af25dfa41053b1f6 not found in new config with opid_index 2) I0913 23:14:25.164070 15122 catalog_manager.cc:3698] Sending ChangeConfig:ADD_PEER:VOTER on tablet f9b2cffa071640fbb75fb8b0430cb6f7 (attempt 1) I0913 23:14:25.164839 15199 tablet_service.cc:1066] Processing DeleteTablet for tablet f9b2cffa071640fbb75fb8b0430cb6f7 with delete_type TABLET_DATA_TOMBSTONED (TS 63ebaa75f74e4547af25dfa41053b1f6 not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:46098 I0913 23:14:25.166642 15277 consensus_queue.cc:206] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "65475b138e8547789c377ceee9656e60" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 32899 } } peers { permanent_uuid: "afe09d31e65f437faa6e61008d02c307" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 44889 } } peers { permanent_uuid: "63ebaa75f74e4547af25dfa41053b1f6" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 40379 } } I0913 23:14:25.169184 15507 ts_tablet_manager.cc:1440] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0913 23:14:25.170236 15344 raft_consensus.cc:1184] T f9b2cffa071640fbb75fb8b0430cb6f7 P afe09d31e65f437faa6e61008d02c307 [term 1 FOLLOWER]: Refusing update from remote peer 65475b138e8547789c377ceee9656e60: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch) W0913 23:14:25.171003 15239 consensus_peers.cc:458] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 -> Peer 63ebaa75f74e4547af25dfa41053b1f6 (127.13.53.1:40379): Couldn't send request to peer 63ebaa75f74e4547af25dfa41053b1f6. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry. I0913 23:14:25.171667 15500 consensus_queue.cc:984] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afe09d31e65f437faa6e61008d02c307" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 44889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0913 23:14:25.175343 15448 raft_consensus.cc:2792] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 63ebaa75f74e4547af25dfa41053b1f6 (127.13.53.1) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "65475b138e8547789c377ceee9656e60" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 32899 } } peers { permanent_uuid: "afe09d31e65f437faa6e61008d02c307" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 44889 } } peers { permanent_uuid: "63ebaa75f74e4547af25dfa41053b1f6" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 40379 } } } I0913 23:14:25.176669 15344 raft_consensus.cc:2792] T f9b2cffa071640fbb75fb8b0430cb6f7 P afe09d31e65f437faa6e61008d02c307 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 63ebaa75f74e4547af25dfa41053b1f6 (127.13.53.1) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "65475b138e8547789c377ceee9656e60" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 32899 } } peers { permanent_uuid: "afe09d31e65f437faa6e61008d02c307" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 44889 } } peers { permanent_uuid: "63ebaa75f74e4547af25dfa41053b1f6" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 40379 } } } I0913 23:14:25.178668 15507 ts_tablet_manager.cc:1453] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1 I0913 23:14:25.179028 15507 log.cc:990] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6: Deleting WAL directory at /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568416417781642-13524/minicluster-data/ts-0-root/wals/f9b2cffa071640fbb75fb8b0430cb6f7 I0913 23:14:25.180392 15106 catalog_manager.cc:3449] TS 63ebaa75f74e4547af25dfa41053b1f6 (127.13.53.1:40379): tablet f9b2cffa071640fbb75fb8b0430cb6f7 (table test-workload [id=93f0414fe23945a1b6acf86c397866f3]) successfully deleted I0913 23:14:25.180402 15108 catalog_manager.cc:3613] ChangeConfig:ADD_PEER:VOTER RPC for tablet f9b2cffa071640fbb75fb8b0430cb6f7 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1) I0913 23:14:25.183044 15122 catalog_manager.cc:4079] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 reported cstate change: config changed from index 2 to 3, VOTER 63ebaa75f74e4547af25dfa41053b1f6 (127.13.53.1) added. New cstate: current_term: 1 leader_uuid: "65475b138e8547789c377ceee9656e60" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "65475b138e8547789c377ceee9656e60" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 32899 } } peers { permanent_uuid: "afe09d31e65f437faa6e61008d02c307" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 44889 } } peers { permanent_uuid: "63ebaa75f74e4547af25dfa41053b1f6" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 40379 } } } Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- d700d56f713b46b48b7b5d90e96765db | 127.13.53.62:44755 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+-------------------+---------+---------- 63ebaa75f74e4547af25dfa41053b1f6 | 127.13.53.1:40379 | HEALTHY | 65475b138e8547789c377ceee9656e60 | 127.13.53.2:32899 | HEALTHY | afe09d31e65f437faa6e61008d02c307 | 127.13.53.3:44889 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 4 server(s) checked Tablet Summary Tablet f9b2cffa071640fbb75fb8b0430cb6f7 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 65475b138e8547789c377ceee9656e60 (127.13.53.2:32899): RUNNING [LEADER] afe09d31e65f437faa6e61008d02c307 (127.13.53.3:44889): RUNNING 63ebaa75f74e4547af25dfa41053b1f6 (127.13.53.1:40379): not running State: FAILED Data state: TABLET_DATA_TOMBSTONED Last status: Deleted tablet blocks from disk All reported replicas are: A = 65475b138e8547789c377ceee9656e60 B = afe09d31e65f437faa6e61008d02c307 C = 63ebaa75f74e4547af25dfa41053b1f6 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 1 | 3 | Yes B | A* B C | 1 | 3 | Yes C | [config not available] | | | Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0913 23:14:25.538816 13524 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0913 23:14:25.625560 15530 ts_tablet_manager.cc:708] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6: Initiating tablet copy from peer 65475b138e8547789c377ceee9656e60 (127.13.53.2:32899) I0913 23:14:25.626044 15530 tablet_copy_client.cc:241] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6: tablet copy: Beginning tablet copy session from remote peer at address 127.13.53.2:32899 I0913 23:14:25.635741 15287 tablet_copy_service.cc:135] P 65475b138e8547789c377ceee9656e60: Received BeginTabletCopySession request for tablet f9b2cffa071640fbb75fb8b0430cb6f7 from peer 63ebaa75f74e4547af25dfa41053b1f6 ({username='slave'} at 127.0.0.1:50110) I0913 23:14:25.636098 15287 tablet_copy_service.cc:156] P 65475b138e8547789c377ceee9656e60: Beginning new tablet copy session on tablet f9b2cffa071640fbb75fb8b0430cb6f7 from peer 63ebaa75f74e4547af25dfa41053b1f6 at {username='slave'} at 127.0.0.1:50110: session id = 63ebaa75f74e4547af25dfa41053b1f6-f9b2cffa071640fbb75fb8b0430cb6f7 I0913 23:14:25.640681 15287 tablet_copy_source_session.cc:202] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60: Tablet Copy: opened 0 blocks and 1 log segments I0913 23:14:25.642949 15530 ts_tablet_manager.cc:1440] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6: Deleting tablet data with delete state TABLET_DATA_COPYING I0913 23:14:25.651923 15530 ts_tablet_manager.cc:1453] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1 I0913 23:14:25.652400 15530 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet f9b2cffa071640fbb75fb8b0430cb6f7. 1 dirs total, 0 dirs full, 0 dirs failed I0913 23:14:25.656652 15530 tablet_copy_client.cc:579] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6: tablet copy: Starting download of 0 data blocks... I0913 23:14:25.657060 15530 tablet_copy_client.cc:542] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6: tablet copy: Starting download of 1 WAL segments... I0913 23:14:25.659878 15530 tablet_copy_client.cc:418] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0913 23:14:25.664954 15530 tablet_bootstrap.cc:439] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6: Bootstrap starting. I0913 23:14:25.675004 15530 tablet_bootstrap.cc:439] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0913 23:14:25.675590 15530 tablet_bootstrap.cc:439] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6: Bootstrap complete. I0913 23:14:25.676028 15530 ts_tablet_manager.cc:1085] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.004s I0913 23:14:25.677352 15530 raft_consensus.cc:340] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "65475b138e8547789c377ceee9656e60" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 32899 } } peers { permanent_uuid: "afe09d31e65f437faa6e61008d02c307" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 44889 } } peers { permanent_uuid: "63ebaa75f74e4547af25dfa41053b1f6" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 40379 } } I0913 23:14:25.677878 15530 raft_consensus.cc:705] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ebaa75f74e4547af25dfa41053b1f6, State: Initialized, Role: FOLLOWER I0913 23:14:25.678414 15530 consensus_queue.cc:229] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "65475b138e8547789c377ceee9656e60" member_type: VOTER last_known_addr { host: "127.13.53.2" port: 32899 } } peers { permanent_uuid: "afe09d31e65f437faa6e61008d02c307" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 44889 } } peers { permanent_uuid: "63ebaa75f74e4547af25dfa41053b1f6" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 40379 } } I0913 23:14:25.680343 15530 ts_tablet_manager.cc:1113] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6: Time spent starting tablet: real 0.004s user 0.002s sys 0.004s I0913 23:14:25.681638 15287 tablet_copy_service.cc:325] P 65475b138e8547789c377ceee9656e60: Request end of tablet copy session 63ebaa75f74e4547af25dfa41053b1f6-f9b2cffa071640fbb75fb8b0430cb6f7 received from {username='slave'} at 127.0.0.1:50110 I0913 23:14:25.682126 15287 tablet_copy_service.cc:413] P 65475b138e8547789c377ceee9656e60: ending tablet copy session 63ebaa75f74e4547af25dfa41053b1f6-f9b2cffa071640fbb75fb8b0430cb6f7 on tablet f9b2cffa071640fbb75fb8b0430cb6f7 with peer 63ebaa75f74e4547af25dfa41053b1f6 Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- d700d56f713b46b48b7b5d90e96765db | 127.13.53.62:44755 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+-------------------+---------+---------- 63ebaa75f74e4547af25dfa41053b1f6 | 127.13.53.1:40379 | HEALTHY | 65475b138e8547789c377ceee9656e60 | 127.13.53.2:32899 | HEALTHY | afe09d31e65f437faa6e61008d02c307 | 127.13.53.3:44889 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 4 server(s) checked Tablet Summary Tablet f9b2cffa071640fbb75fb8b0430cb6f7 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's 65475b138e8547789c377ceee9656e60 (127.13.53.2:32899): RUNNING [LEADER] afe09d31e65f437faa6e61008d02c307 (127.13.53.3:44889): RUNNING 63ebaa75f74e4547af25dfa41053b1f6 (127.13.53.1:40379): RUNNING All reported replicas are: A = 65475b138e8547789c377ceee9656e60 B = afe09d31e65f437faa6e61008d02c307 C = 63ebaa75f74e4547af25dfa41053b1f6 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 1 | 3 | Yes B | A* B C | 1 | 3 | Yes C | A B C | 1 | 3 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0913 23:14:25.911206 13524 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0913 23:14:26.168252 15208 raft_consensus.cc:1126] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[] Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- d700d56f713b46b48b7b5d90e96765db | 127.13.53.62:44755 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+-------------------+---------+---------- 63ebaa75f74e4547af25dfa41053b1f6 | 127.13.53.1:40379 | HEALTHY | 65475b138e8547789c377ceee9656e60 | 127.13.53.2:32899 | HEALTHY | afe09d31e65f437faa6e61008d02c307 | 127.13.53.3:44889 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 4 server(s) checked Tablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+---------+---------------+---------+------------+------------------+------------- test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0913 23:14:26.398802 13524 log_verifier.cc:121] Checking tablet f9b2cffa071640fbb75fb8b0430cb6f7 W0913 23:14:26.461927 15309 debug-util.cc:397] Leaking SignalData structure 0x7b08001cd820 after lost signal to thread 13527 I0913 23:14:26.463167 13524 log_verifier.cc:172] Verified matching terms for 3 ops in tablet f9b2cffa071640fbb75fb8b0430cb6f7 I0913 23:14:26.465070 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:14:26.491374 13524 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0913 23:14:26.491649 13524 tablet_replica.cc:273] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6: stopping tablet replica I0913 23:14:26.491968 13524 raft_consensus.cc:2147] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6 [term 1 FOLLOWER]: Raft consensus shutting down. I0913 23:14:26.492226 13524 raft_consensus.cc:2174] T f9b2cffa071640fbb75fb8b0430cb6f7 P 63ebaa75f74e4547af25dfa41053b1f6 [term 1 FOLLOWER]: Raft consensus is shut down! I0913 23:14:26.512655 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:14:26.512876 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:14:26.513270 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:14:26.520834 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:14:26.539227 13524 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0913 23:14:26.539578 13524 tablet_replica.cc:273] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60: stopping tablet replica I0913 23:14:26.540030 13524 raft_consensus.cc:2147] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 [term 1 LEADER]: Raft consensus shutting down. I0913 23:14:26.540877 13524 raft_consensus.cc:2174] T f9b2cffa071640fbb75fb8b0430cb6f7 P 65475b138e8547789c377ceee9656e60 [term 1 FOLLOWER]: Raft consensus is shut down! I0913 23:14:26.561883 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:14:26.562038 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:14:26.562379 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:14:26.568969 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:14:26.592434 13524 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0913 23:14:26.592736 13524 tablet_replica.cc:273] T f9b2cffa071640fbb75fb8b0430cb6f7 P afe09d31e65f437faa6e61008d02c307: stopping tablet replica I0913 23:14:26.593055 13524 raft_consensus.cc:2147] T f9b2cffa071640fbb75fb8b0430cb6f7 P afe09d31e65f437faa6e61008d02c307 [term 1 FOLLOWER]: Raft consensus shutting down. I0913 23:14:26.593334 13524 raft_consensus.cc:2174] T f9b2cffa071640fbb75fb8b0430cb6f7 P afe09d31e65f437faa6e61008d02c307 [term 1 FOLLOWER]: Raft consensus is shut down! W0913 23:14:27.611356 13524 thread.cc:499] Waited for 1000ms trying to join with diag-logger (tid 15309) W0913 23:14:28.611986 13524 thread.cc:499] Waited for 2000ms trying to join with diag-logger (tid 15309) I0913 23:14:29.077005 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:14:29.077221 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:14:29.077700 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:14:29.087206 13524 master.cc:247] Master@127.13.53.62:44755 shutting down... I0913 23:14:29.109791 13524 raft_consensus.cc:2147] T 00000000000000000000000000000000 P d700d56f713b46b48b7b5d90e96765db [term 1 LEADER]: Raft consensus shutting down. I0913 23:14:29.110263 13524 raft_consensus.cc:2174] T 00000000000000000000000000000000 P d700d56f713b46b48b7b5d90e96765db [term 1 FOLLOWER]: Raft consensus is shut down! I0913 23:14:29.110503 13524 tablet_replica.cc:273] T 00000000000000000000000000000000 P d700d56f713b46b48b7b5d90e96765db: stopping tablet replica I0913 23:14:29.127861 13524 master.cc:258] Master@127.13.53.62:44755 shutdown complete. [ OK ] FailedTabletsAreReplacedITest.OneReplica/0 (6012 ms) [ RUN ] FailedTabletsAreReplacedITest.OneReplica/1 I0913 23:14:29.152269 13524 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.13.53.62:39937 I0913 23:14:29.163154 13524 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 39700us I0913 23:14:29.163472 13524 fs_manager.cc:263] Metadata directory not provided I0913 23:14:29.163581 13524 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0913 23:14:29.163748 13524 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0913 23:14:29.163847 13524 server_base.cc:449] Attempting to create new FS layout instead I0913 23:14:29.164786 13524 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568416417781642-13524/minicluster-data/master-0-root/instance: uuid: "310c552b96bb462192f548dd9b70f13b" format_stamp: "Formatted at 2019-09-13 23:14:29 on dist-test-slave-dist-test-slave-kz3c" I0913 23:14:29.172663 13524 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.006s sys 0.004s I0913 23:14:29.173990 13524 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0913 23:14:29.180105 13524 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0913 23:14:29.180250 13524 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568416417781642-13524/minicluster-data/master-0-root uuid: "310c552b96bb462192f548dd9b70f13b" format_stamp: "Formatted at 2019-09-13 23:14:29 on dist-test-slave-dist-test-slave-kz3c" I0913 23:14:29.180447 13524 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568416417781642-13524/minicluster-data/master-0-root metadata directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568416417781642-13524/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568416417781642-13524/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0913 23:14:29.258463 13524 rpc_server.cc:205] RPC server started. Bound to: 127.13.53.62:39937 I0913 23:14:29.260545 13524 webserver.cc:374] Webserver started at http://127.13.53.62:32777/ using document root and password file I0913 23:14:29.262528 15637 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0913 23:14:29.270972 15637 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 310c552b96bb462192f548dd9b70f13b: Bootstrap starting. I0913 23:14:29.273612 15637 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 310c552b96bb462192f548dd9b70f13b: No blocks or log segments found. Creating new log. I0913 23:14:29.276847 15637 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 310c552b96bb462192f548dd9b70f13b: No bootstrap required, opened a new log I0913 23:14:29.278345 15637 raft_consensus.cc:340] T 00000000000000000000000000000000 P 310c552b96bb462192f548dd9b70f13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310c552b96bb462192f548dd9b70f13b" member_type: VOTER } I0913 23:14:29.278703 15637 raft_consensus.cc:366] T 00000000000000000000000000000000 P 310c552b96bb462192f548dd9b70f13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0913 23:14:29.278853 15637 raft_consensus.cc:705] T 00000000000000000000000000000000 P 310c552b96bb462192f548dd9b70f13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 310c552b96bb462192f548dd9b70f13b, State: Initialized, Role: FOLLOWER I0913 23:14:29.279295 15637 consensus_queue.cc:229] T 00000000000000000000000000000000 P 310c552b96bb462192f548dd9b70f13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310c552b96bb462192f548dd9b70f13b" member_type: VOTER } I0913 23:14:29.279690 15637 raft_consensus.cc:380] T 00000000000000000000000000000000 P 310c552b96bb462192f548dd9b70f13b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0913 23:14:29.279835 15637 raft_consensus.cc:472] T 00000000000000000000000000000000 P 310c552b96bb462192f548dd9b70f13b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0913 23:14:29.280021 15637 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 310c552b96bb462192f548dd9b70f13b [term 0 FOLLOWER]: Advancing to term 1 I0913 23:14:29.283390 15637 raft_consensus.cc:494] T 00000000000000000000000000000000 P 310c552b96bb462192f548dd9b70f13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310c552b96bb462192f548dd9b70f13b" member_type: VOTER } I0913 23:14:29.283766 15637 leader_election.cc:310] T 00000000000000000000000000000000 P 310c552b96bb462192f548dd9b70f13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 310c552b96bb462192f548dd9b70f13b; no voters: I0913 23:14:29.285017 15639 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 310c552b96bb462192f548dd9b70f13b [term 1 FOLLOWER]: Leader election won for term 1 I0913 23:14:29.285694 15637 leader_election.cc:296] T 00000000000000000000000000000000 P 310c552b96bb462192f548dd9b70f13b [CANDIDATE]: Term 1 election: Requested vote from peers I0913 23:14:29.287258 15639 raft_consensus.cc:667] T 00000000000000000000000000000000 P 310c552b96bb462192f548dd9b70f13b [term 1 LEADER]: Becoming Leader. State: Replica: 310c552b96bb462192f548dd9b70f13b, State: Running, Role: LEADER I0913 23:14:29.287628 15639 consensus_queue.cc:206] T 00000000000000000000000000000000 P 310c552b96bb462192f548dd9b70f13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310c552b96bb462192f548dd9b70f13b" member_type: VOTER } I0913 23:14:29.291347 15641 sys_catalog.cc:340] T 00000000000000000000000000000000 P 310c552b96bb462192f548dd9b70f13b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 310c552b96bb462192f548dd9b70f13b. Latest consensus state: current_term: 1 leader_uuid: "310c552b96bb462192f548dd9b70f13b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310c552b96bb462192f548dd9b70f13b" member_type: VOTER } } I0913 23:14:29.292131 15641 sys_catalog.cc:343] T 00000000000000000000000000000000 P 310c552b96bb462192f548dd9b70f13b [sys.catalog]: This master's current role is: LEADER I0913 23:14:29.292311 15637 sys_catalog.cc:340] T 00000000000000000000000000000000 P 310c552b96bb462192f548dd9b70f13b [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "310c552b96bb462192f548dd9b70f13b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310c552b96bb462192f548dd9b70f13b" member_type: VOTER } } I0913 23:14:29.292853 15637 sys_catalog.cc:343] T 00000000000000000000000000000000 P 310c552b96bb462192f548dd9b70f13b [sys.catalog]: This master's current role is: LEADER I0913 23:14:29.293107 15640 sys_catalog.cc:340] T 00000000000000000000000000000000 P 310c552b96bb462192f548dd9b70f13b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "310c552b96bb462192f548dd9b70f13b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310c552b96bb462192f548dd9b70f13b" member_type: VOTER } } I0913 23:14:29.293488 15640 sys_catalog.cc:343] T 00000000000000000000000000000000 P 310c552b96bb462192f548dd9b70f13b [sys.catalog]: This master's current role is: LEADER I0913 23:14:29.294241 15637 sys_catalog.cc:428] T 00000000000000000000000000000000 P 310c552b96bb462192f548dd9b70f13b [sys.catalog]: configured and running, proceeding with master startup. I0913 23:14:29.295853 15645 catalog_manager.cc:1071] Loading table and tablet metadata into memory... I0913 23:14:29.298612 15646 catalog_manager.cc:554] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting I0913 23:14:29.298923 15645 catalog_manager.cc:1082] Initializing Kudu internal certificate authority... I0913 23:14:29.298956 13524 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0913 23:14:29.327708 15645 catalog_manager.cc:966] Generated new certificate authority record I0913 23:14:29.329147 15645 catalog_manager.cc:1091] Loading token signing keys... I0913 23:14:29.338250 15645 catalog_manager.cc:4394] T 00000000000000000000000000000000 P 310c552b96bb462192f548dd9b70f13b: Generated new TSK 0 I0913 23:14:29.375408 13524 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 39700us I0913 23:14:29.375789 13524 fs_manager.cc:263] Metadata directory not provided I0913 23:14:29.375931 13524 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0913 23:14:29.376112 13524 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0913 23:14:29.376214 13524 server_base.cc:449] Attempting to create new FS layout instead I0913 23:14:29.377120 13524 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568416417781642-13524/minicluster-data/ts-0-root/instance: uuid: "c83bcad877994240b4c1d9d7b8730d0a" format_stamp: "Formatted at 2019-09-13 23:14:29 on dist-test-slave-dist-test-slave-kz3c" I0913 23:14:29.385303 13524 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.009s sys 0.001s I0913 23:14:29.386772 13524 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0913 23:14:29.393903 13524 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0913 23:14:29.394110 13524 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568416417781642-13524/minicluster-data/ts-0-root uuid: "c83bcad877994240b4c1d9d7b8730d0a" format_stamp: "Formatted at 2019-09-13 23:14:29 on dist-test-slave-dist-test-slave-kz3c" I0913 23:14:29.394322 13524 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568416417781642-13524/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568416417781642-13524/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568416417781642-13524/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0913 23:14:29.443703 13524 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0913 23:14:29.443863 13524 ts_tablet_manager.cc:403] Registered 0 tablets I0913 23:14:29.481899 13524 rpc_server.cc:205] RPC server started. Bound to: 127.13.53.1:37639 I0913 23:14:29.483850 13524 webserver.cc:374] Webserver started at http://127.13.53.1:35041/ using document root and password file I0913 23:14:29.502523 15712 heartbeater.cc:322] Connected to a master server at 127.13.53.62:39937 I0913 23:14:29.502800 15712 heartbeater.cc:406] Registering TS with master... I0913 23:14:29.503214 15712 heartbeater.cc:455] Master 127.13.53.62:39937 requested a full tablet report, sending... I0913 23:14:29.503341 13524 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 39700us I0913 23:14:29.503728 13524 fs_manager.cc:263] Metadata directory not provided I0913 23:14:29.503904 13524 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0913 23:14:29.504133 13524 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0913 23:14:29.504237 13524 server_base.cc:449] Attempting to create new FS layout instead I0913 23:14:29.504865 15604 ts_manager.cc:161] Registered new tserver with Master: c83bcad877994240b4c1d9d7b8730d0a (127.13.53.1:37639) I0913 23:14:29.505178 13524 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568416417781642-13524/minicluster-data/ts-1-root/instance: uuid: "018e408365bc4b3ea92b48d27d2a0240" format_stamp: "Formatted at 2019-09-13 23:14:29 on dist-test-slave-dist-test-slave-kz3c" I0913 23:14:29.506736 15604 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34708 I0913 23:14:29.513490 13524 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s I0913 23:14:29.514962 13524 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0913 23:14:29.521697 13524 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0913 23:14:29.521864 13524 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568416417781642-13524/minicluster-data/ts-1-root uuid: "018e408365bc4b3ea92b48d27d2a0240" format_stamp: "Formatted at 2019-09-13 23:14:29 on dist-test-slave-dist-test-slave-kz3c" I0913 23:14:29.522120 13524 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568416417781642-13524/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568416417781642-13524/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568416417781642-13524/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0913 23:14:29.564294 13524 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0913 23:14:29.564471 13524 ts_tablet_manager.cc:403] Registered 0 tablets I0913 23:14:29.601923 13524 rpc_server.cc:205] RPC server started. Bound to: 127.13.53.2:40441 I0913 23:14:29.603648 13524 webserver.cc:374] Webserver started at http://127.13.53.2:39865/ using document root and password file I0913 23:14:29.615039 15780 heartbeater.cc:322] Connected to a master server at 127.13.53.62:39937 I0913 23:14:29.615296 15780 heartbeater.cc:406] Registering TS with master... I0913 23:14:29.615686 15780 heartbeater.cc:455] Master 127.13.53.62:39937 requested a full tablet report, sending... I0913 23:14:29.617040 15604 ts_manager.cc:161] Registered new tserver with Master: 018e408365bc4b3ea92b48d27d2a0240 (127.13.53.2:40441) I0913 23:14:29.618325 15604 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34710 I0913 23:14:29.619675 13524 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 39700us I0913 23:14:29.619992 13524 fs_manager.cc:263] Metadata directory not provided I0913 23:14:29.620124 13524 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0913 23:14:29.620293 13524 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0913 23:14:29.620407 13524 server_base.cc:449] Attempting to create new FS layout instead I0913 23:14:29.621253 13524 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568416417781642-13524/minicluster-data/ts-2-root/instance: uuid: "e13c1fc70bd848da8d9eaaac021b296e" format_stamp: "Formatted at 2019-09-13 23:14:29 on dist-test-slave-dist-test-slave-kz3c" I0913 23:14:29.629928 13524 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.011s sys 0.000s I0913 23:14:29.631227 13524 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0913 23:14:29.638183 13524 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0913 23:14:29.638347 13524 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568416417781642-13524/minicluster-data/ts-2-root uuid: "e13c1fc70bd848da8d9eaaac021b296e" format_stamp: "Formatted at 2019-09-13 23:14:29 on dist-test-slave-dist-test-slave-kz3c" I0913 23:14:29.638579 13524 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568416417781642-13524/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568416417781642-13524/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568416417781642-13524/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0913 23:14:29.659255 13524 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0913 23:14:29.659430 13524 ts_tablet_manager.cc:403] Registered 0 tablets I0913 23:14:29.696144 13524 rpc_server.cc:205] RPC server started. Bound to: 127.13.53.3:32973 I0913 23:14:29.697898 13524 webserver.cc:374] Webserver started at http://127.13.53.3:39549/ using document root and password file I0913 23:14:29.710197 15847 heartbeater.cc:322] Connected to a master server at 127.13.53.62:39937 I0913 23:14:29.710471 15847 heartbeater.cc:406] Registering TS with master... I0913 23:14:29.710860 15847 heartbeater.cc:455] Master 127.13.53.62:39937 requested a full tablet report, sending... I0913 23:14:29.712198 15604 ts_manager.cc:161] Registered new tserver with Master: e13c1fc70bd848da8d9eaaac021b296e (127.13.53.3:32973) I0913 23:14:29.713399 15604 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34712 I0913 23:14:29.716024 13524 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 39700us I0913 23:14:29.716346 13524 fs_manager.cc:263] Metadata directory not provided I0913 23:14:29.716490 13524 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0913 23:14:29.716663 13524 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0913 23:14:29.716769 13524 server_base.cc:449] Attempting to create new FS layout instead I0913 23:14:29.717661 13524 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568416417781642-13524/minicluster-data/ts-3-root/instance: uuid: "8bdf2a1759a84503a0bf482a1e931d06" format_stamp: "Formatted at 2019-09-13 23:14:29 on dist-test-slave-dist-test-slave-kz3c" I0913 23:14:29.727166 13524 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.009s sys 0.002s I0913 23:14:29.728612 13524 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0913 23:14:29.736078 13524 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0913 23:14:29.736261 13524 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568416417781642-13524/minicluster-data/ts-3-root uuid: "8bdf2a1759a84503a0bf482a1e931d06" format_stamp: "Formatted at 2019-09-13 23:14:29 on dist-test-slave-dist-test-slave-kz3c" I0913 23:14:29.736543 13524 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568416417781642-13524/minicluster-data/ts-3-root metadata directory: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568416417781642-13524/minicluster-data/ts-3-root 1 data directories: /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568416417781642-13524/minicluster-data/ts-3-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0913 23:14:29.773465 13524 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0913 23:14:29.773658 13524 ts_tablet_manager.cc:403] Registered 0 tablets I0913 23:14:29.811146 13524 rpc_server.cc:205] RPC server started. Bound to: 127.13.53.4:43881 I0913 23:14:29.812878 13524 webserver.cc:374] Webserver started at http://127.13.53.4:43313/ using document root and password file I0913 23:14:29.823946 15914 heartbeater.cc:322] Connected to a master server at 127.13.53.62:39937 I0913 23:14:29.824259 15914 heartbeater.cc:406] Registering TS with master... I0913 23:14:29.824630 15914 heartbeater.cc:455] Master 127.13.53.62:39937 requested a full tablet report, sending... I0913 23:14:29.825899 15604 ts_manager.cc:161] Registered new tserver with Master: 8bdf2a1759a84503a0bf482a1e931d06 (127.13.53.4:43881) I0913 23:14:29.826411 13524 internal_mini_cluster.cc:328] 4 TS(s) registered with all masters after 0.011974474s I0913 23:14:29.827229 15604 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34714 I0913 23:14:29.844221 13524 test_util.cc:212] Using random seed: 1787527397 I0913 23:14:29.861079 15604 catalog_manager.cc:1425] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34718: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0913 23:14:29.904814 15679 tablet_service.cc:1022] Processing CreateTablet for tablet 02f46be75a3a4d36aee92ef7e5176503 (table=test-workload [id=2517b33ad97f4be5b6571d6968d57859]), partition=RANGE (key) PARTITION UNBOUNDED I0913 23:14:29.905836 15679 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 02f46be75a3a4d36aee92ef7e5176503. 1 dirs total, 0 dirs full, 0 dirs failed I0913 23:14:29.904814 15881 tablet_service.cc:1022] Processing CreateTablet for tablet 02f46be75a3a4d36aee92ef7e5176503 (table=test-workload [id=2517b33ad97f4be5b6571d6968d57859]), partition=RANGE (key) PARTITION UNBOUNDED I0913 23:14:29.906087 15814 tablet_service.cc:1022] Processing CreateTablet for tablet 02f46be75a3a4d36aee92ef7e5176503 (table=test-workload [id=2517b33ad97f4be5b6571d6968d57859]), partition=RANGE (key) PARTITION UNBOUNDED I0913 23:14:29.906576 15881 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 02f46be75a3a4d36aee92ef7e5176503. 1 dirs total, 0 dirs full, 0 dirs failed I0913 23:14:29.906790 15814 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 02f46be75a3a4d36aee92ef7e5176503. 1 dirs total, 0 dirs full, 0 dirs failed I0913 23:14:29.922765 15938 tablet_bootstrap.cc:439] T 02f46be75a3a4d36aee92ef7e5176503 P c83bcad877994240b4c1d9d7b8730d0a: Bootstrap starting. I0913 23:14:29.923904 15937 tablet_bootstrap.cc:439] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e: Bootstrap starting. I0913 23:14:29.925281 15939 tablet_bootstrap.cc:439] T 02f46be75a3a4d36aee92ef7e5176503 P 8bdf2a1759a84503a0bf482a1e931d06: Bootstrap starting. I0913 23:14:29.927042 15938 tablet_bootstrap.cc:590] T 02f46be75a3a4d36aee92ef7e5176503 P c83bcad877994240b4c1d9d7b8730d0a: No blocks or log segments found. Creating new log. I0913 23:14:29.927570 15937 tablet_bootstrap.cc:590] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e: No blocks or log segments found. Creating new log. I0913 23:14:29.928634 15939 tablet_bootstrap.cc:590] T 02f46be75a3a4d36aee92ef7e5176503 P 8bdf2a1759a84503a0bf482a1e931d06: No blocks or log segments found. Creating new log. I0913 23:14:29.932162 15937 tablet_bootstrap.cc:439] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e: No bootstrap required, opened a new log I0913 23:14:29.932502 15937 ts_tablet_manager.cc:1085] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s I0913 23:14:29.932528 15938 tablet_bootstrap.cc:439] T 02f46be75a3a4d36aee92ef7e5176503 P c83bcad877994240b4c1d9d7b8730d0a: No bootstrap required, opened a new log I0913 23:14:29.933240 15938 ts_tablet_manager.cc:1085] T 02f46be75a3a4d36aee92ef7e5176503 P c83bcad877994240b4c1d9d7b8730d0a: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s I0913 23:14:29.934557 15939 tablet_bootstrap.cc:439] T 02f46be75a3a4d36aee92ef7e5176503 P 8bdf2a1759a84503a0bf482a1e931d06: No bootstrap required, opened a new log I0913 23:14:29.934494 15937 raft_consensus.cc:340] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83bcad877994240b4c1d9d7b8730d0a" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 37639 } } peers { permanent_uuid: "8bdf2a1759a84503a0bf482a1e931d06" member_type: VOTER last_known_addr { host: "127.13.53.4" port: 43881 } } peers { permanent_uuid: "e13c1fc70bd848da8d9eaaac021b296e" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 32973 } } I0913 23:14:29.934995 15939 ts_tablet_manager.cc:1085] T 02f46be75a3a4d36aee92ef7e5176503 P 8bdf2a1759a84503a0bf482a1e931d06: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s I0913 23:14:29.935092 15937 raft_consensus.cc:366] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0913 23:14:29.935534 15937 raft_consensus.cc:705] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e13c1fc70bd848da8d9eaaac021b296e, State: Initialized, Role: FOLLOWER I0913 23:14:29.935048 15938 raft_consensus.cc:340] T 02f46be75a3a4d36aee92ef7e5176503 P c83bcad877994240b4c1d9d7b8730d0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83bcad877994240b4c1d9d7b8730d0a" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 37639 } } peers { permanent_uuid: "8bdf2a1759a84503a0bf482a1e931d06" member_type: VOTER last_known_addr { host: "127.13.53.4" port: 43881 } } peers { permanent_uuid: "e13c1fc70bd848da8d9eaaac021b296e" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 32973 } } I0913 23:14:29.935968 15938 raft_consensus.cc:366] T 02f46be75a3a4d36aee92ef7e5176503 P c83bcad877994240b4c1d9d7b8730d0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0913 23:14:29.936241 15938 raft_consensus.cc:705] T 02f46be75a3a4d36aee92ef7e5176503 P c83bcad877994240b4c1d9d7b8730d0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c83bcad877994240b4c1d9d7b8730d0a, State: Initialized, Role: FOLLOWER I0913 23:14:29.936005 15937 consensus_queue.cc:229] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83bcad877994240b4c1d9d7b8730d0a" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 37639 } } peers { permanent_uuid: "8bdf2a1759a84503a0bf482a1e931d06" member_type: VOTER last_known_addr { host: "127.13.53.4" port: 43881 } } peers { permanent_uuid: "e13c1fc70bd848da8d9eaaac021b296e" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 32973 } } I0913 23:14:29.936938 15939 raft_consensus.cc:340] T 02f46be75a3a4d36aee92ef7e5176503 P 8bdf2a1759a84503a0bf482a1e931d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83bcad877994240b4c1d9d7b8730d0a" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 37639 } } peers { permanent_uuid: "8bdf2a1759a84503a0bf482a1e931d06" member_type: VOTER last_known_addr { host: "127.13.53.4" port: 43881 } } peers { permanent_uuid: "e13c1fc70bd848da8d9eaaac021b296e" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 32973 } } I0913 23:14:29.936847 15938 consensus_queue.cc:229] T 02f46be75a3a4d36aee92ef7e5176503 P c83bcad877994240b4c1d9d7b8730d0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83bcad877994240b4c1d9d7b8730d0a" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 37639 } } peers { permanent_uuid: "8bdf2a1759a84503a0bf482a1e931d06" member_type: VOTER last_known_addr { host: "127.13.53.4" port: 43881 } } peers { permanent_uuid: "e13c1fc70bd848da8d9eaaac021b296e" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 32973 } } I0913 23:14:29.937546 15939 raft_consensus.cc:366] T 02f46be75a3a4d36aee92ef7e5176503 P 8bdf2a1759a84503a0bf482a1e931d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0913 23:14:29.938236 15939 raft_consensus.cc:705] T 02f46be75a3a4d36aee92ef7e5176503 P 8bdf2a1759a84503a0bf482a1e931d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bdf2a1759a84503a0bf482a1e931d06, State: Initialized, Role: FOLLOWER I0913 23:14:29.939682 15712 heartbeater.cc:447] Master 127.13.53.62:39937 was elected leader, sending a full tablet report... I0913 23:14:29.939909 15938 ts_tablet_manager.cc:1113] T 02f46be75a3a4d36aee92ef7e5176503 P c83bcad877994240b4c1d9d7b8730d0a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s I0913 23:14:29.939942 15847 heartbeater.cc:447] Master 127.13.53.62:39937 was elected leader, sending a full tablet report... I0913 23:14:29.940217 15937 ts_tablet_manager.cc:1113] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s I0913 23:14:29.939173 15939 consensus_queue.cc:229] T 02f46be75a3a4d36aee92ef7e5176503 P 8bdf2a1759a84503a0bf482a1e931d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83bcad877994240b4c1d9d7b8730d0a" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 37639 } } peers { permanent_uuid: "8bdf2a1759a84503a0bf482a1e931d06" member_type: VOTER last_known_addr { host: "127.13.53.4" port: 43881 } } peers { permanent_uuid: "e13c1fc70bd848da8d9eaaac021b296e" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 32973 } } I0913 23:14:29.943578 15914 heartbeater.cc:447] Master 127.13.53.62:39937 was elected leader, sending a full tablet report... I0913 23:14:29.944842 15939 ts_tablet_manager.cc:1113] T 02f46be75a3a4d36aee92ef7e5176503 P 8bdf2a1759a84503a0bf482a1e931d06: Time spent starting tablet: real 0.009s user 0.001s sys 0.005s I0913 23:14:30.023944 15943 raft_consensus.cc:472] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0913 23:14:30.024408 15943 raft_consensus.cc:494] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83bcad877994240b4c1d9d7b8730d0a" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 37639 } } peers { permanent_uuid: "8bdf2a1759a84503a0bf482a1e931d06" member_type: VOTER last_known_addr { host: "127.13.53.4" port: 43881 } } peers { permanent_uuid: "e13c1fc70bd848da8d9eaaac021b296e" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 32973 } } I0913 23:14:30.025770 15943 leader_election.cc:296] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c83bcad877994240b4c1d9d7b8730d0a (127.13.53.1:37639), 8bdf2a1759a84503a0bf482a1e931d06 (127.13.53.4:43881) I0913 23:14:30.036377 15689 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "02f46be75a3a4d36aee92ef7e5176503" candidate_uuid: "e13c1fc70bd848da8d9eaaac021b296e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c83bcad877994240b4c1d9d7b8730d0a" is_pre_election: true I0913 23:14:30.037127 15689 raft_consensus.cc:2350] T 02f46be75a3a4d36aee92ef7e5176503 P c83bcad877994240b4c1d9d7b8730d0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e13c1fc70bd848da8d9eaaac021b296e in term 0. I0913 23:14:30.038166 15787 leader_election.cc:310] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c83bcad877994240b4c1d9d7b8730d0a, e13c1fc70bd848da8d9eaaac021b296e; no voters: I0913 23:14:30.038028 15891 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "02f46be75a3a4d36aee92ef7e5176503" candidate_uuid: "e13c1fc70bd848da8d9eaaac021b296e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bdf2a1759a84503a0bf482a1e931d06" is_pre_election: true I0913 23:14:30.038848 15891 raft_consensus.cc:2350] T 02f46be75a3a4d36aee92ef7e5176503 P 8bdf2a1759a84503a0bf482a1e931d06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e13c1fc70bd848da8d9eaaac021b296e in term 0. I0913 23:14:30.039010 15943 raft_consensus.cc:2644] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e [term 0 FOLLOWER]: Leader pre-election won for term 1 I0913 23:14:30.039305 15943 raft_consensus.cc:472] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0913 23:14:30.039516 15943 raft_consensus.cc:2890] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e [term 0 FOLLOWER]: Advancing to term 1 I0913 23:14:30.043468 15943 raft_consensus.cc:494] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83bcad877994240b4c1d9d7b8730d0a" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 37639 } } peers { permanent_uuid: "8bdf2a1759a84503a0bf482a1e931d06" member_type: VOTER last_known_addr { host: "127.13.53.4" port: 43881 } } peers { permanent_uuid: "e13c1fc70bd848da8d9eaaac021b296e" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 32973 } } I0913 23:14:30.044829 15943 leader_election.cc:296] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e [CANDIDATE]: Term 1 election: Requested vote from peers c83bcad877994240b4c1d9d7b8730d0a (127.13.53.1:37639), 8bdf2a1759a84503a0bf482a1e931d06 (127.13.53.4:43881) I0913 23:14:30.045423 15689 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "02f46be75a3a4d36aee92ef7e5176503" candidate_uuid: "e13c1fc70bd848da8d9eaaac021b296e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c83bcad877994240b4c1d9d7b8730d0a" I0913 23:14:30.045660 15891 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "02f46be75a3a4d36aee92ef7e5176503" candidate_uuid: "e13c1fc70bd848da8d9eaaac021b296e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bdf2a1759a84503a0bf482a1e931d06" I0913 23:14:30.046002 15689 raft_consensus.cc:2890] T 02f46be75a3a4d36aee92ef7e5176503 P c83bcad877994240b4c1d9d7b8730d0a [term 0 FOLLOWER]: Advancing to term 1 I0913 23:14:30.046169 15891 raft_consensus.cc:2890] T 02f46be75a3a4d36aee92ef7e5176503 P 8bdf2a1759a84503a0bf482a1e931d06 [term 0 FOLLOWER]: Advancing to term 1 I0913 23:14:30.051489 15891 raft_consensus.cc:2350] T 02f46be75a3a4d36aee92ef7e5176503 P 8bdf2a1759a84503a0bf482a1e931d06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e13c1fc70bd848da8d9eaaac021b296e in term 1. I0913 23:14:30.051489 15689 raft_consensus.cc:2350] T 02f46be75a3a4d36aee92ef7e5176503 P c83bcad877994240b4c1d9d7b8730d0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e13c1fc70bd848da8d9eaaac021b296e in term 1. I0913 23:14:30.052561 15787 leader_election.cc:310] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c83bcad877994240b4c1d9d7b8730d0a, e13c1fc70bd848da8d9eaaac021b296e; no voters: I0913 23:14:30.053308 15943 raft_consensus.cc:2644] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e [term 1 FOLLOWER]: Leader election won for term 1 I0913 23:14:30.054477 15943 raft_consensus.cc:667] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e [term 1 LEADER]: Becoming Leader. State: Replica: e13c1fc70bd848da8d9eaaac021b296e, State: Running, Role: LEADER I0913 23:14:30.054951 15943 consensus_queue.cc:206] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83bcad877994240b4c1d9d7b8730d0a" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 37639 } } peers { permanent_uuid: "8bdf2a1759a84503a0bf482a1e931d06" member_type: VOTER last_known_addr { host: "127.13.53.4" port: 43881 } } peers { permanent_uuid: "e13c1fc70bd848da8d9eaaac021b296e" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 32973 } } I0913 23:14:30.059684 15603 catalog_manager.cc:4079] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e reported cstate change: term changed from 0 to 1, leader changed from to e13c1fc70bd848da8d9eaaac021b296e (127.13.53.3). New cstate: current_term: 1 leader_uuid: "e13c1fc70bd848da8d9eaaac021b296e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83bcad877994240b4c1d9d7b8730d0a" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 37639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bdf2a1759a84503a0bf482a1e931d06" member_type: VOTER last_known_addr { host: "127.13.53.4" port: 43881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e13c1fc70bd848da8d9eaaac021b296e" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 32973 } health_report { overall_health: HEALTHY } } } Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- 310c552b96bb462192f548dd9b70f13b | 127.13.53.62:39937 | HEALTHY Flag | Value | Tags | Master ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+-------------------+---------+---------- 018e408365bc4b3ea92b48d27d2a0240 | 127.13.53.2:40441 | HEALTHY | 8bdf2a1759a84503a0bf482a1e931d06 | 127.13.53.4:43881 | HEALTHY | c83bcad877994240b4c1d9d7b8730d0a | 127.13.53.1:37639 | HEALTHY | e13c1fc70bd848da8d9eaaac021b296e | 127.13.53.3:32973 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet 02f46be75a3a4d36aee92ef7e5176503 of table 'test-workload' is conflicted: 5 replicas' active configs disagree with the leader master's c83bcad877994240b4c1d9d7b8730d0a (127.13.53.1:37639): RUNNING 8bdf2a1759a84503a0bf482a1e931d06 (127.13.53.4:43881): RUNNING e13c1fc70bd848da8d9eaaac021b296e (127.13.53.3:32973): RUNNING [LEADER] All reported replicas are: A = c83bcad877994240b4c1d9d7b8730d0a B = 8bdf2a1759a84503a0bf482a1e931d06 C = e13c1fc70bd848da8d9eaaac021b296e The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C | 1 | -1 | Yes B | A B C | 1 | -1 | Yes C | A B C* | 1 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 0 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+-------------------+--------------- Small | 018e408365bc4b3ea92b48d27d2a0240 | 127.13.53.2:40441 | 0 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0913 23:14:30.262300 13524 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0913 23:14:30.478425 15943 consensus_queue.cc:984] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bdf2a1759a84503a0bf482a1e931d06" member_type: VOTER last_known_addr { host: "127.13.53.4" port: 43881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0913 23:14:30.495405 15943 consensus_queue.cc:984] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c83bcad877994240b4c1d9d7b8730d0a" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 37639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- 310c552b96bb462192f548dd9b70f13b | 127.13.53.62:39937 | HEALTHY Flag | Value | Tags | Master ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+-------------------+---------+---------- 018e408365bc4b3ea92b48d27d2a0240 | 127.13.53.2:40441 | HEALTHY | 8bdf2a1759a84503a0bf482a1e931d06 | 127.13.53.4:43881 | HEALTHY | c83bcad877994240b4c1d9d7b8730d0a | 127.13.53.1:37639 | HEALTHY | e13c1fc70bd848da8d9eaaac021b296e | 127.13.53.3:32973 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 5 server(s) checked Tablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+---------+---------------+---------+------------+------------------+------------- test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 0 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+-------------------+--------------- Small | 018e408365bc4b3ea92b48d27d2a0240 | 127.13.53.2:40441 | 0 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0913 23:14:30.592526 13524 log_verifier.cc:121] Checking tablet 02f46be75a3a4d36aee92ef7e5176503 I0913 23:14:30.620806 15780 heartbeater.cc:447] Master 127.13.53.62:39937 was elected leader, sending a full tablet report... I0913 23:14:30.660663 13524 log_verifier.cc:172] Verified matching terms for 1 ops in tablet 02f46be75a3a4d36aee92ef7e5176503 I0913 23:14:30.661123 13524 tablet_replica.cc:273] T 02f46be75a3a4d36aee92ef7e5176503 P 8bdf2a1759a84503a0bf482a1e931d06: stopping tablet replica I0913 23:14:30.661465 13524 raft_consensus.cc:2147] T 02f46be75a3a4d36aee92ef7e5176503 P 8bdf2a1759a84503a0bf482a1e931d06 [term 1 FOLLOWER]: Raft consensus shutting down. I0913 23:14:30.661739 13524 raft_consensus.cc:2174] T 02f46be75a3a4d36aee92ef7e5176503 P 8bdf2a1759a84503a0bf482a1e931d06 [term 1 FOLLOWER]: Raft consensus is shut down! Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- 310c552b96bb462192f548dd9b70f13b | 127.13.53.62:39937 | HEALTHY Flag | Value | Tags | Master ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+-------------------+---------+---------- 018e408365bc4b3ea92b48d27d2a0240 | 127.13.53.2:40441 | HEALTHY | 8bdf2a1759a84503a0bf482a1e931d06 | 127.13.53.4:43881 | HEALTHY | c83bcad877994240b4c1d9d7b8730d0a | 127.13.53.1:37639 | HEALTHY | e13c1fc70bd848da8d9eaaac021b296e | 127.13.53.3:32973 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet 02f46be75a3a4d36aee92ef7e5176503 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING c83bcad877994240b4c1d9d7b8730d0a (127.13.53.1:37639): RUNNING 8bdf2a1759a84503a0bf482a1e931d06 (127.13.53.4:43881): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: INJECTED ERROR: tablet failed e13c1fc70bd848da8d9eaaac021b296e (127.13.53.3:32973): RUNNING [LEADER] All reported replicas are: A = c83bcad877994240b4c1d9d7b8730d0a B = 8bdf2a1759a84503a0bf482a1e931d06 C = e13c1fc70bd848da8d9eaaac021b296e The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C* | 1 | -1 | Yes B | [config not available] | | | C | A B C* | 1 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 0 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+-------------------+--------------- Small | 018e408365bc4b3ea92b48d27d2a0240 | 127.13.53.2:40441 | 0 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0913 23:14:30.820034 13524 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0913 23:14:30.985394 15785 consensus_queue.cc:544] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e [LEADER]: The tablet replica hosted on peer 8bdf2a1759a84503a0bf482a1e931d06 has failed W0913 23:14:30.985783 15785 consensus_peers.cc:458] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e -> Peer 8bdf2a1759a84503a0bf482a1e931d06 (127.13.53.4:43881): Couldn't send request to peer 8bdf2a1759a84503a0bf482a1e931d06. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry. I0913 23:14:31.029603 15604 catalog_manager.cc:3830] Sending ChangeConfig:REMOVE_PEER on tablet 02f46be75a3a4d36aee92ef7e5176503 (attempt 1) I0913 23:14:31.031975 15824 consensus_queue.cc:206] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c83bcad877994240b4c1d9d7b8730d0a" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 37639 } } peers { permanent_uuid: "e13c1fc70bd848da8d9eaaac021b296e" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 32973 } } I0913 23:14:31.035686 15689 raft_consensus.cc:1184] T 02f46be75a3a4d36aee92ef7e5176503 P c83bcad877994240b4c1d9d7b8730d0a [term 1 FOLLOWER]: Refusing update from remote peer e13c1fc70bd848da8d9eaaac021b296e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0913 23:14:31.036751 16039 consensus_queue.cc:984] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c83bcad877994240b4c1d9d7b8730d0a" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 37639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0913 23:14:31.041079 15943 raft_consensus.cc:2792] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8bdf2a1759a84503a0bf482a1e931d06 (127.13.53.4) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c83bcad877994240b4c1d9d7b8730d0a" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 37639 } } peers { permanent_uuid: "e13c1fc70bd848da8d9eaaac021b296e" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 32973 } } } I0913 23:14:31.042117 15689 raft_consensus.cc:2792] T 02f46be75a3a4d36aee92ef7e5176503 P c83bcad877994240b4c1d9d7b8730d0a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8bdf2a1759a84503a0bf482a1e931d06 (127.13.53.4) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c83bcad877994240b4c1d9d7b8730d0a" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 37639 } } peers { permanent_uuid: "e13c1fc70bd848da8d9eaaac021b296e" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 32973 } } } I0913 23:14:31.047963 15585 catalog_manager.cc:3613] ChangeConfig:REMOVE_PEER RPC for tablet 02f46be75a3a4d36aee92ef7e5176503 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0913 23:14:31.050859 15604 catalog_manager.cc:4079] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e reported cstate change: config changed from index -1 to 2, VOTER 8bdf2a1759a84503a0bf482a1e931d06 (127.13.53.4) evicted. New cstate: current_term: 1 leader_uuid: "e13c1fc70bd848da8d9eaaac021b296e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c83bcad877994240b4c1d9d7b8730d0a" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 37639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e13c1fc70bd848da8d9eaaac021b296e" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 32973 } health_report { overall_health: HEALTHY } } } I0913 23:14:31.061623 15604 catalog_manager.cc:3471] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 02f46be75a3a4d36aee92ef7e5176503 on 8bdf2a1759a84503a0bf482a1e931d06 (127.13.53.4:43881) (TS 8bdf2a1759a84503a0bf482a1e931d06 not found in new config with opid_index 2) I0913 23:14:31.062309 15604 catalog_manager.cc:3698] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 02f46be75a3a4d36aee92ef7e5176503 (attempt 1) I0913 23:14:31.062927 15881 tablet_service.cc:1066] Processing DeleteTablet for tablet 02f46be75a3a4d36aee92ef7e5176503 with delete_type TABLET_DATA_TOMBSTONED (TS 8bdf2a1759a84503a0bf482a1e931d06 not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:50272 I0913 23:14:31.065011 15824 consensus_queue.cc:206] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c83bcad877994240b4c1d9d7b8730d0a" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 37639 } } peers { permanent_uuid: "e13c1fc70bd848da8d9eaaac021b296e" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 32973 } } peers { permanent_uuid: "018e408365bc4b3ea92b48d27d2a0240" member_type: NON_VOTER last_known_addr { host: "127.13.53.2" port: 40441 } attrs { promote: true } } I0913 23:14:31.076516 15689 raft_consensus.cc:1184] T 02f46be75a3a4d36aee92ef7e5176503 P c83bcad877994240b4c1d9d7b8730d0a [term 1 FOLLOWER]: Refusing update from remote peer e13c1fc70bd848da8d9eaaac021b296e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I0913 23:14:31.078131 16039 consensus_queue.cc:984] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c83bcad877994240b4c1d9d7b8730d0a" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 37639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s I0913 23:14:31.079331 16049 ts_tablet_manager.cc:1440] T 02f46be75a3a4d36aee92ef7e5176503 P 8bdf2a1759a84503a0bf482a1e931d06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0913 23:14:31.083782 15943 raft_consensus.cc:2792] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 018e408365bc4b3ea92b48d27d2a0240 (127.13.53.2) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c83bcad877994240b4c1d9d7b8730d0a" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 37639 } } peers { permanent_uuid: "e13c1fc70bd848da8d9eaaac021b296e" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 32973 } } peers { permanent_uuid: "018e408365bc4b3ea92b48d27d2a0240" member_type: NON_VOTER last_known_addr { host: "127.13.53.2" port: 40441 } attrs { promote: true } } } W0913 23:14:31.088462 15787 consensus_peers.cc:458] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e -> Peer 018e408365bc4b3ea92b48d27d2a0240 (127.13.53.2:40441): Couldn't send request to peer 018e408365bc4b3ea92b48d27d2a0240. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02f46be75a3a4d36aee92ef7e5176503. This is attempt 1: this message will repeat every 5th retry. I0913 23:14:31.089582 15689 raft_consensus.cc:2792] T 02f46be75a3a4d36aee92ef7e5176503 P c83bcad877994240b4c1d9d7b8730d0a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 018e408365bc4b3ea92b48d27d2a0240 (127.13.53.2) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c83bcad877994240b4c1d9d7b8730d0a" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 37639 } } peers { permanent_uuid: "e13c1fc70bd848da8d9eaaac021b296e" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 32973 } } peers { permanent_uuid: "018e408365bc4b3ea92b48d27d2a0240" member_type: NON_VOTER last_known_addr { host: "127.13.53.2" port: 40441 } attrs { promote: true } } } I0913 23:14:31.093412 15585 catalog_manager.cc:3613] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 02f46be75a3a4d36aee92ef7e5176503 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0913 23:14:31.094295 16049 ts_tablet_manager.cc:1453] T 02f46be75a3a4d36aee92ef7e5176503 P 8bdf2a1759a84503a0bf482a1e931d06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1 I0913 23:14:31.094844 16049 log.cc:990] T 02f46be75a3a4d36aee92ef7e5176503 P 8bdf2a1759a84503a0bf482a1e931d06: Deleting WAL directory at /tmp/dist-test-taskMj6sdl/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568416417781642-13524/minicluster-data/ts-3-root/wals/02f46be75a3a4d36aee92ef7e5176503 I0913 23:14:31.096352 15585 catalog_manager.cc:3449] TS 8bdf2a1759a84503a0bf482a1e931d06 (127.13.53.4:43881): tablet 02f46be75a3a4d36aee92ef7e5176503 (table test-workload [id=2517b33ad97f4be5b6571d6968d57859]) successfully deleted I0913 23:14:31.097589 15604 catalog_manager.cc:4079] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e reported cstate change: config changed from index 2 to 3, NON_VOTER 018e408365bc4b3ea92b48d27d2a0240 (127.13.53.2) added. New cstate: current_term: 1 leader_uuid: "e13c1fc70bd848da8d9eaaac021b296e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c83bcad877994240b4c1d9d7b8730d0a" member_type: VOTER last_known_addr { host: "127.13.53.1" port: 37639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e13c1fc70bd848da8d9eaaac021b296e" member_type: VOTER last_known_addr { host: "127.13.53.3" port: 32973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "018e408365bc4b3ea92b48d27d2a0240" member_type: NON_VOTER last_known_addr { host: "127.13.53.2" port: 40441 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- 310c552b96bb462192f548dd9b70f13b | 127.13.53.62:39937 | HEALTHY Flag | Value | Tags | Master ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+-------------------+---------+---------- 018e408365bc4b3ea92b48d27d2a0240 | 127.13.53.2:40441 | HEALTHY | 8bdf2a1759a84503a0bf482a1e931d06 | 127.13.53.4:43881 | HEALTHY | c83bcad877994240b4c1d9d7b8730d0a | 127.13.53.1:37639 | HEALTHY | e13c1fc70bd848da8d9eaaac021b296e | 127.13.53.3:32973 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 5 server(s) checked Tablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+---------+---------------+---------+------------+------------------+------------- test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 0 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+-------------------+--------------- Small | 8bdf2a1759a84503a0bf482a1e931d06 | 127.13.53.4:43881 | 0 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0913 23:14:31.212378 13524 log_verifier.cc:121] Checking tablet 02f46be75a3a4d36aee92ef7e5176503 I0913 23:14:31.256134 13524 log_verifier.cc:172] Verified matching terms for 3 ops in tablet 02f46be75a3a4d36aee92ef7e5176503 I0913 23:14:31.257479 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:14:31.280267 13524 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0913 23:14:31.280571 13524 tablet_replica.cc:273] T 02f46be75a3a4d36aee92ef7e5176503 P c83bcad877994240b4c1d9d7b8730d0a: stopping tablet replica I0913 23:14:31.280905 13524 raft_consensus.cc:2147] T 02f46be75a3a4d36aee92ef7e5176503 P c83bcad877994240b4c1d9d7b8730d0a [term 1 FOLLOWER]: Raft consensus shutting down. I0913 23:14:31.281221 13524 raft_consensus.cc:2174] T 02f46be75a3a4d36aee92ef7e5176503 P c83bcad877994240b4c1d9d7b8730d0a [term 1 FOLLOWER]: Raft consensus is shut down! I0913 23:14:31.300288 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:14:31.300467 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:14:31.300773 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:14:31.307143 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:14:31.322402 13524 ts_tablet_manager.cc:1159] Shutting down tablet manager... W0913 23:14:31.591130 15787 consensus_peers.cc:436] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e -> Peer 018e408365bc4b3ea92b48d27d2a0240 (127.13.53.2:40441): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.53.2:40441: connect: Connection refused (error 111) W0913 23:14:31.666903 15787 consensus_peers.cc:458] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e -> Peer c83bcad877994240b4c1d9d7b8730d0a (127.13.53.1:37639): Couldn't send request to peer c83bcad877994240b4c1d9d7b8730d0a. Status: Network error: Client connection negotiation failed: client connection to 127.13.53.1:37639: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. W0913 23:14:31.935029 15722 debug-util.cc:397] Leaking SignalData structure 0x7b080018ea60 after lost signal to thread 13527 W0913 23:14:32.088253 15787 consensus_peers.cc:436] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e -> Peer 018e408365bc4b3ea92b48d27d2a0240 (127.13.53.2:40441): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.53.2:40441: connect: Connection refused (error 111) W0913 23:14:32.335896 13524 thread.cc:499] Waited for 1000ms trying to join with diag-logger (tid 15722) W0913 23:14:32.595675 15787 consensus_peers.cc:436] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e -> Peer 018e408365bc4b3ea92b48d27d2a0240 (127.13.53.2:40441): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.53.2:40441: connect: Connection refused (error 111) W0913 23:14:33.195211 15787 consensus_peers.cc:436] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e -> Peer 018e408365bc4b3ea92b48d27d2a0240 (127.13.53.2:40441): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.53.2:40441: connect: Connection refused (error 111) W0913 23:14:33.336474 13524 thread.cc:499] Waited for 2000ms trying to join with diag-logger (tid 15722) W0913 23:14:33.765924 15787 consensus_peers.cc:436] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e -> Peer 018e408365bc4b3ea92b48d27d2a0240 (127.13.53.2:40441): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.13.53.2:40441: connect: Connection refused (error 111) I0913 23:14:33.913439 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:14:33.913615 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:14:33.913960 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:14:33.919643 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:14:33.937029 13524 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0913 23:14:33.937369 13524 tablet_replica.cc:273] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e: stopping tablet replica I0913 23:14:33.937680 13524 raft_consensus.cc:2147] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e [term 1 LEADER]: Raft consensus shutting down. I0913 23:14:33.938293 13524 raft_consensus.cc:2174] T 02f46be75a3a4d36aee92ef7e5176503 P e13c1fc70bd848da8d9eaaac021b296e [term 1 FOLLOWER]: Raft consensus is shut down! I0913 23:14:33.956251 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:14:33.956418 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:14:33.956722 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:14:33.962949 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:14:33.979230 13524 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0913 23:14:33.995508 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:14:33.995674 13524 tablet_server.cc:155] TabletServer shutting down... I0913 23:14:33.995954 13524 tablet_server.cc:169] TabletServer shutdown complete. I0913 23:14:34.002213 13524 master.cc:247] Master@127.13.53.62:39937 shutting down... I0913 23:14:34.015362 13524 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 310c552b96bb462192f548dd9b70f13b [term 1 LEADER]: Raft consensus shutting down. I0913 23:14:34.015673 13524 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 310c552b96bb462192f548dd9b70f13b [term 1 FOLLOWER]: Raft consensus is shut down! I0913 23:14:34.015871 13524 tablet_replica.cc:273] T 00000000000000000000000000000000 P 310c552b96bb462192f548dd9b70f13b: stopping tablet replica I0913 23:14:34.031658 13524 master.cc:258] Master@127.13.53.62:39937 shutdown complete. [ OK ] FailedTabletsAreReplacedITest.OneReplica/1 (4899 ms) [----------] 2 tests from FailedTabletsAreReplacedITest (10911 ms total) [----------] Global test environment tear-down [==========] 6 tests from 2 test cases ran. (56186 ms total) [ PASSED ] 5 tests. [ FAILED ] 1 test, listed below: [ FAILED ] TsTabletManagerITest.TestTableStats 1 FAILED TEST