[==========] Running 6 tests from 2 test cases. [----------] Global test environment set-up. [----------] 4 tests from TsTabletManagerITest [ RUN ] TsTabletManagerITest.TestReportNewLeaderOnLeaderChange WARNING: Logging before InitGoogleLogging() is written to STDERR I0917 09:19:38.789041 14145 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.13.208.126:42631 I0917 09:19:38.790531 14145 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 21804us I0917 09:19:38.790652 14145 fs_manager.cc:263] Metadata directory not provided I0917 09:19:38.790684 14145 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0917 09:19:38.790750 14145 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0917 09:19:38.790771 14145 server_base.cc:453] Attempting to create new FS layout instead I0917 09:19:38.791817 14145 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568711978778636-14145/minicluster-data/master-0-root/instance: uuid: "5bbc635a9c7e42e9af0179b15b50ec45" format_stamp: "Formatted at 2019-09-17 09:19:38 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:38.793234 14145 fs_manager.cc:503] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s I0917 09:19:38.793524 14145 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0917 09:19:38.793634 14145 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0917 09:19:38.794236 14145 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:38.794284 14145 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568711978778636-14145/minicluster-data/master-0-root uuid: "5bbc635a9c7e42e9af0179b15b50ec45" format_stamp: "Formatted at 2019-09-17 09:19:38 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:38.794332 14145 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568711978778636-14145/minicluster-data/master-0-root metadata directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568711978778636-14145/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568711978778636-14145/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0917 09:19:38.819397 14145 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0917 09:19:38.822939 14145 rpc_server.cc:205] RPC server started. Bound to: 127.13.208.126:42631 I0917 09:19:38.823223 14145 webserver.cc:374] Webserver started at http://127.13.208.126:41047/ using document root and password file I0917 09:19:38.823652 14208 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0917 09:19:38.826896 14208 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 5bbc635a9c7e42e9af0179b15b50ec45: Bootstrap starting. I0917 09:19:38.827435 14208 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 5bbc635a9c7e42e9af0179b15b50ec45: No blocks or log segments found. Creating new log. I0917 09:19:38.827837 14208 log.cc:525] T 00000000000000000000000000000000 P 5bbc635a9c7e42e9af0179b15b50ec45: Log is configured to *not* fsync() on all Append() calls I0917 09:19:38.828305 14208 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 5bbc635a9c7e42e9af0179b15b50ec45: No bootstrap required, opened a new log I0917 09:19:38.830222 14208 raft_consensus.cc:340] T 00000000000000000000000000000000 P 5bbc635a9c7e42e9af0179b15b50ec45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbc635a9c7e42e9af0179b15b50ec45" member_type: VOTER } I0917 09:19:38.830262 14208 raft_consensus.cc:366] T 00000000000000000000000000000000 P 5bbc635a9c7e42e9af0179b15b50ec45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0917 09:19:38.830273 14208 raft_consensus.cc:705] T 00000000000000000000000000000000 P 5bbc635a9c7e42e9af0179b15b50ec45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bbc635a9c7e42e9af0179b15b50ec45, State: Initialized, Role: FOLLOWER I0917 09:19:38.830456 14208 consensus_queue.cc:229] T 00000000000000000000000000000000 P 5bbc635a9c7e42e9af0179b15b50ec45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbc635a9c7e42e9af0179b15b50ec45" member_type: VOTER } I0917 09:19:38.830479 14208 raft_consensus.cc:380] T 00000000000000000000000000000000 P 5bbc635a9c7e42e9af0179b15b50ec45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0917 09:19:38.830490 14208 raft_consensus.cc:472] T 00000000000000000000000000000000 P 5bbc635a9c7e42e9af0179b15b50ec45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0917 09:19:38.830505 14208 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 5bbc635a9c7e42e9af0179b15b50ec45 [term 0 FOLLOWER]: Advancing to term 1 I0917 09:19:38.831059 14208 raft_consensus.cc:494] T 00000000000000000000000000000000 P 5bbc635a9c7e42e9af0179b15b50ec45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbc635a9c7e42e9af0179b15b50ec45" member_type: VOTER } I0917 09:19:38.831135 14208 leader_election.cc:310] T 00000000000000000000000000000000 P 5bbc635a9c7e42e9af0179b15b50ec45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bbc635a9c7e42e9af0179b15b50ec45; no voters: I0917 09:19:38.831251 14208 leader_election.cc:296] T 00000000000000000000000000000000 P 5bbc635a9c7e42e9af0179b15b50ec45 [CANDIDATE]: Term 1 election: Requested vote from peers I0917 09:19:38.831365 14208 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5bbc635a9c7e42e9af0179b15b50ec45 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbc635a9c7e42e9af0179b15b50ec45" member_type: VOTER } } I0917 09:19:38.831384 14208 sys_catalog.cc:343] T 00000000000000000000000000000000 P 5bbc635a9c7e42e9af0179b15b50ec45 [sys.catalog]: This master's current role is: FOLLOWER I0917 09:19:38.831490 14208 sys_catalog.cc:428] T 00000000000000000000000000000000 P 5bbc635a9c7e42e9af0179b15b50ec45 [sys.catalog]: configured and running, proceeding with master startup. I0917 09:19:38.831610 14210 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 5bbc635a9c7e42e9af0179b15b50ec45 [term 1 FOLLOWER]: Leader election won for term 1 I0917 09:19:38.831614 14145 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0917 09:19:38.831678 14210 raft_consensus.cc:667] T 00000000000000000000000000000000 P 5bbc635a9c7e42e9af0179b15b50ec45 [term 1 LEADER]: Becoming Leader. State: Replica: 5bbc635a9c7e42e9af0179b15b50ec45, State: Running, Role: LEADER I0917 09:19:38.831743 14210 consensus_queue.cc:206] T 00000000000000000000000000000000 P 5bbc635a9c7e42e9af0179b15b50ec45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbc635a9c7e42e9af0179b15b50ec45" member_type: VOTER } I0917 09:19:38.831981 14210 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5bbc635a9c7e42e9af0179b15b50ec45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bbc635a9c7e42e9af0179b15b50ec45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbc635a9c7e42e9af0179b15b50ec45" member_type: VOTER } } I0917 09:19:38.832000 14210 sys_catalog.cc:343] T 00000000000000000000000000000000 P 5bbc635a9c7e42e9af0179b15b50ec45 [sys.catalog]: This master's current role is: LEADER I0917 09:19:38.832063 14210 sys_catalog.cc:340] T 00000000000000000000000000000000 P 5bbc635a9c7e42e9af0179b15b50ec45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bbc635a9c7e42e9af0179b15b50ec45. Latest consensus state: current_term: 1 leader_uuid: "5bbc635a9c7e42e9af0179b15b50ec45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bbc635a9c7e42e9af0179b15b50ec45" member_type: VOTER } } I0917 09:19:38.832077 14210 sys_catalog.cc:343] T 00000000000000000000000000000000 P 5bbc635a9c7e42e9af0179b15b50ec45 [sys.catalog]: This master's current role is: LEADER W0917 09:19:38.832432 14212 catalog_manager.cc:1144] T 00000000000000000000000000000000 P 5bbc635a9c7e42e9af0179b15b50ec45: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0917 09:19:38.832448 14212 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0917 09:19:38.833019 14215 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0917 09:19:38.833185 14215 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0917 09:19:38.885295 14215 catalog_manager.cc:981] Generated new certificate authority record I0917 09:19:38.885926 14215 catalog_manager.cc:1106] Loading token signing keys... I0917 09:19:38.895246 14215 catalog_manager.cc:4435] T 00000000000000000000000000000000 P 5bbc635a9c7e42e9af0179b15b50ec45: Generated new TSK 0 I0917 09:19:38.896971 14145 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 21804us I0917 09:19:38.897043 14145 fs_manager.cc:263] Metadata directory not provided I0917 09:19:38.897069 14145 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0917 09:19:38.897109 14145 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0917 09:19:38.897120 14145 server_base.cc:453] Attempting to create new FS layout instead I0917 09:19:38.897369 14145 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568711978778636-14145/minicluster-data/ts-0-root/instance: uuid: "3b4eceaa1f8246de8fd9a3309a08848b" format_stamp: "Formatted at 2019-09-17 09:19:38 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:38.898478 14145 fs_manager.cc:503] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s I0917 09:19:38.898712 14145 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0917 09:19:38.899394 14145 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:38.899430 14145 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568711978778636-14145/minicluster-data/ts-0-root uuid: "3b4eceaa1f8246de8fd9a3309a08848b" format_stamp: "Formatted at 2019-09-17 09:19:38 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:38.899449 14145 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568711978778636-14145/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568711978778636-14145/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568711978778636-14145/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0917 09:19:38.906045 14145 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0917 09:19:38.906080 14145 ts_tablet_manager.cc:403] Registered 0 tablets I0917 09:19:38.909394 14145 rpc_server.cc:205] RPC server started. Bound to: 127.13.208.65:35707 I0917 09:19:38.909595 14145 webserver.cc:374] Webserver started at http://127.13.208.65:42941/ using document root and password file I0917 09:19:38.910885 14145 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 21804us I0917 09:19:38.910949 14145 fs_manager.cc:263] Metadata directory not provided I0917 09:19:38.910989 14145 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0917 09:19:38.911029 14145 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0917 09:19:38.911039 14145 server_base.cc:453] Attempting to create new FS layout instead I0917 09:19:38.911265 14145 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568711978778636-14145/minicluster-data/ts-1-root/instance: uuid: "7e80b3cd6d95403f8499e84cc013b527" format_stamp: "Formatted at 2019-09-17 09:19:38 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:38.913444 14145 fs_manager.cc:503] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s I0917 09:19:38.913794 14145 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0917 09:19:38.914624 14145 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:38.914670 14145 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568711978778636-14145/minicluster-data/ts-1-root uuid: "7e80b3cd6d95403f8499e84cc013b527" format_stamp: "Formatted at 2019-09-17 09:19:38 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:38.914690 14145 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568711978778636-14145/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568711978778636-14145/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestReportNewLeaderOnLeaderChange.1568711978778636-14145/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0917 09:19:38.922106 14283 heartbeater.cc:322] Connected to a master server at 127.13.208.126:42631 I0917 09:19:38.922174 14283 heartbeater.cc:406] Registering TS with master... I0917 09:19:38.922283 14283 heartbeater.cc:455] Master 127.13.208.126:42631 requested a full tablet report, sending... I0917 09:19:38.922693 14171 ts_manager.cc:161] Registered new tserver with Master: 3b4eceaa1f8246de8fd9a3309a08848b (127.13.208.65:35707) I0917 09:19:38.923763 14171 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54298 I0917 09:19:38.949569 14145 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0917 09:19:38.949595 14145 ts_tablet_manager.cc:403] Registered 0 tablets I0917 09:19:38.952322 14145 rpc_server.cc:205] RPC server started. Bound to: 127.13.208.66:40627 I0917 09:19:38.952569 14145 webserver.cc:374] Webserver started at http://127.13.208.66:44881/ using document root and password file I0917 09:19:38.955757 14351 heartbeater.cc:322] Connected to a master server at 127.13.208.126:42631 I0917 09:19:38.955796 14351 heartbeater.cc:406] Registering TS with master... I0917 09:19:38.955847 14351 heartbeater.cc:455] Master 127.13.208.126:42631 requested a full tablet report, sending... I0917 09:19:38.956044 14171 ts_manager.cc:161] Registered new tserver with Master: 7e80b3cd6d95403f8499e84cc013b527 (127.13.208.66:40627) I0917 09:19:38.956465 14171 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54300 I0917 09:19:38.957085 14145 internal_mini_cluster.cc:328] 2 TS(s) registered with all masters after 0.00436355s I0917 09:19:38.964166 14171 catalog_manager.cc:1440] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54302: name: "test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 2 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0917 09:19:38.964344 14171 catalog_manager.cc:1605] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers. I0917 09:19:38.969642 14250 tablet_service.cc:1022] Processing CreateTablet for tablet 252b39981f3b42368cd835c1f62536e5 (table=test-table [id=7f2aeb18ebca40418e6f98dadeef1617]), partition=RANGE (key) PARTITION UNBOUNDED I0917 09:19:38.969784 14250 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 252b39981f3b42368cd835c1f62536e5. 1 dirs total, 0 dirs full, 0 dirs failed I0917 09:19:38.970155 14318 tablet_service.cc:1022] Processing CreateTablet for tablet 252b39981f3b42368cd835c1f62536e5 (table=test-table [id=7f2aeb18ebca40418e6f98dadeef1617]), partition=RANGE (key) PARTITION UNBOUNDED I0917 09:19:38.970244 14318 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 252b39981f3b42368cd835c1f62536e5. 1 dirs total, 0 dirs full, 0 dirs failed I0917 09:19:38.971231 14364 tablet_bootstrap.cc:439] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b: Bootstrap starting. I0917 09:19:38.971295 14365 tablet_bootstrap.cc:439] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527: Bootstrap starting. I0917 09:19:38.971669 14364 tablet_bootstrap.cc:590] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b: No blocks or log segments found. Creating new log. I0917 09:19:38.971688 14365 tablet_bootstrap.cc:590] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527: No blocks or log segments found. Creating new log. I0917 09:19:38.972374 14365 tablet_bootstrap.cc:439] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527: No bootstrap required, opened a new log I0917 09:19:38.972383 14364 tablet_bootstrap.cc:439] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b: No bootstrap required, opened a new log I0917 09:19:38.972429 14364 ts_tablet_manager.cc:1085] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I0917 09:19:38.972429 14365 ts_tablet_manager.cc:1085] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I0917 09:19:38.972674 14365 raft_consensus.cc:340] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e80b3cd6d95403f8499e84cc013b527" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40627 } } peers { permanent_uuid: "3b4eceaa1f8246de8fd9a3309a08848b" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35707 } } I0917 09:19:38.972674 14364 raft_consensus.cc:340] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e80b3cd6d95403f8499e84cc013b527" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40627 } } peers { permanent_uuid: "3b4eceaa1f8246de8fd9a3309a08848b" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35707 } } I0917 09:19:38.972702 14365 raft_consensus.cc:705] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e80b3cd6d95403f8499e84cc013b527, State: Initialized, Role: FOLLOWER I0917 09:19:38.972705 14364 raft_consensus.cc:705] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b4eceaa1f8246de8fd9a3309a08848b, State: Initialized, Role: FOLLOWER I0917 09:19:38.972748 14364 consensus_queue.cc:229] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e80b3cd6d95403f8499e84cc013b527" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40627 } } peers { permanent_uuid: "3b4eceaa1f8246de8fd9a3309a08848b" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35707 } } I0917 09:19:38.972748 14365 consensus_queue.cc:229] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e80b3cd6d95403f8499e84cc013b527" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40627 } } peers { permanent_uuid: "3b4eceaa1f8246de8fd9a3309a08848b" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35707 } } I0917 09:19:38.972976 14365 ts_tablet_manager.cc:1113] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I0917 09:19:38.973001 14351 heartbeater.cc:447] Master 127.13.208.126:42631 was elected leader, sending a full tablet report... I0917 09:19:38.972972 14283 heartbeater.cc:447] Master 127.13.208.126:42631 was elected leader, sending a full tablet report... I0917 09:19:38.972977 14364 ts_tablet_manager.cc:1113] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I0917 09:19:38.978420 14145 ts_tablet_manager-itest.cc:324] Electing peer 0... I0917 09:19:38.978471 14145 raft_consensus.cc:405] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 0 FOLLOWER]: Emulating election... I0917 09:19:38.978478 14145 raft_consensus.cc:2890] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 0 FOLLOWER]: Advancing to term 1 I0917 09:19:38.979133 14145 raft_consensus.cc:667] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 1 LEADER]: Becoming Leader. State: Replica: 3b4eceaa1f8246de8fd9a3309a08848b, State: Running, Role: LEADER W0917 09:19:38.979213 14283 heartbeater.cc:571] Failed to heartbeat to 127.13.208.126:42631 (0 consecutive failures): IO error: failing all heartbeats for tests I0917 09:19:38.979221 14145 consensus_queue.cc:206] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e80b3cd6d95403f8499e84cc013b527" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40627 } } peers { permanent_uuid: "3b4eceaa1f8246de8fd9a3309a08848b" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35707 } } I0917 09:19:38.979455 14145 ts_tablet_manager-itest.cc:327] Waiting for servers to agree... I0917 09:19:38.984571 14145 cluster_itest_util.cc:259] Not converged past 1 yet: 1.1 0.0 I0917 09:19:39.085592 14145 cluster_itest_util.cc:259] Not converged past 1 yet: 1.1 0.0 W0917 09:19:39.163465 14284 tablet.cc:1763] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b: Can't schedule compaction. Clean time has not been advanced past its initial value. I0917 09:19:39.286697 14145 cluster_itest_util.cc:259] Not converged past 1 yet: 1.1 0.0 I0917 09:19:39.441895 14328 raft_consensus.cc:2890] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 0 FOLLOWER]: Advancing to term 1 I0917 09:19:39.442921 14368 consensus_queue.cc:984] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e80b3cd6d95403f8499e84cc013b527" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0917 09:19:39.587937 14145 ts_tablet_manager-itest.cc:324] Electing peer 1... I0917 09:19:39.587975 14145 raft_consensus.cc:405] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 1 FOLLOWER]: Emulating election... I0917 09:19:39.587981 14145 raft_consensus.cc:2890] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 1 FOLLOWER]: Advancing to term 2 I0917 09:19:39.588702 14145 raft_consensus.cc:667] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 2 LEADER]: Becoming Leader. State: Replica: 7e80b3cd6d95403f8499e84cc013b527, State: Running, Role: LEADER I0917 09:19:39.588804 14145 consensus_queue.cc:206] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e80b3cd6d95403f8499e84cc013b527" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40627 } } peers { permanent_uuid: "3b4eceaa1f8246de8fd9a3309a08848b" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35707 } } I0917 09:19:39.589059 14145 ts_tablet_manager-itest.cc:327] Waiting for servers to agree... I0917 09:19:39.589696 14145 cluster_itest_util.cc:259] Not converged past 2 yet: 1.1 2.2 I0917 09:19:39.690631 14145 cluster_itest_util.cc:259] Not converged past 2 yet: 1.1 2.2 I0917 09:19:39.845125 14328 raft_consensus.cc:1149] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 2 LEADER]: Rejecting Update request from peer 3b4eceaa1f8246de8fd9a3309a08848b for earlier term 1. Current term is 2. Ops: [] I0917 09:19:39.845451 14374 consensus_queue.cc:995] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e80b3cd6d95403f8499e84cc013b527" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40627 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0917 09:19:39.845494 14374 consensus_queue.cc:1162] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [LEADER]: Peer 7e80b3cd6d95403f8499e84cc013b527 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1 I0917 09:19:39.845516 14374 raft_consensus.cc:2885] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 1 LEADER]: Stepping down as leader of term 1 I0917 09:19:39.845530 14374 raft_consensus.cc:705] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3b4eceaa1f8246de8fd9a3309a08848b, State: Running, Role: LEADER I0917 09:19:39.845582 14374 consensus_queue.cc:229] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e80b3cd6d95403f8499e84cc013b527" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40627 } } peers { permanent_uuid: "3b4eceaa1f8246de8fd9a3309a08848b" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35707 } } I0917 09:19:39.845646 14374 raft_consensus.cc:2890] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 1 FOLLOWER]: Advancing to term 2 I0917 09:19:39.891455 14145 cluster_itest_util.cc:259] Not converged past 2 yet: 1.1 2.2 I0917 09:19:40.142688 14260 raft_consensus.cc:1184] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 2 FOLLOWER]: Refusing update from remote peer 7e80b3cd6d95403f8499e84cc013b527: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch) I0917 09:19:40.142961 14377 consensus_queue.cc:984] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b4eceaa1f8246de8fd9a3309a08848b" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0917 09:19:40.192438 14145 ts_tablet_manager-itest.cc:324] Electing peer 1... I0917 09:19:40.192471 14145 raft_consensus.cc:405] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 2 LEADER]: Emulating election... I0917 09:19:40.192476 14145 raft_consensus.cc:2885] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 2 LEADER]: Stepping down as leader of term 2 I0917 09:19:40.192482 14145 raft_consensus.cc:705] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7e80b3cd6d95403f8499e84cc013b527, State: Running, Role: LEADER I0917 09:19:40.192567 14145 consensus_queue.cc:229] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e80b3cd6d95403f8499e84cc013b527" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40627 } } peers { permanent_uuid: "3b4eceaa1f8246de8fd9a3309a08848b" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35707 } } I0917 09:19:40.192600 14145 raft_consensus.cc:2890] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 2 FOLLOWER]: Advancing to term 3 I0917 09:19:40.193295 14145 raft_consensus.cc:667] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 3 LEADER]: Becoming Leader. State: Replica: 7e80b3cd6d95403f8499e84cc013b527, State: Running, Role: LEADER I0917 09:19:40.193374 14145 consensus_queue.cc:206] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e80b3cd6d95403f8499e84cc013b527" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40627 } } peers { permanent_uuid: "3b4eceaa1f8246de8fd9a3309a08848b" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35707 } } I0917 09:19:40.193487 14145 ts_tablet_manager-itest.cc:327] Waiting for servers to agree... I0917 09:19:40.194208 14145 cluster_itest_util.cc:259] Not converged past 3 yet: 2.2 3.3 I0917 09:19:40.295073 14145 cluster_itest_util.cc:259] Not converged past 3 yet: 2.2 3.3 I0917 09:19:40.495924 14145 cluster_itest_util.cc:259] Not converged past 3 yet: 2.2 3.3 I0917 09:19:40.638432 14260 raft_consensus.cc:2890] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 2 FOLLOWER]: Advancing to term 3 I0917 09:19:40.639297 14260 raft_consensus.cc:1184] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 3 FOLLOWER]: Refusing update from remote peer 7e80b3cd6d95403f8499e84cc013b527: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch) I0917 09:19:40.639657 14380 consensus_queue.cc:984] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b4eceaa1f8246de8fd9a3309a08848b" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0917 09:19:40.797000 14145 ts_tablet_manager-itest.cc:324] Electing peer 0... I0917 09:19:40.797037 14145 raft_consensus.cc:405] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 3 FOLLOWER]: Emulating election... I0917 09:19:40.797042 14145 raft_consensus.cc:2890] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 3 FOLLOWER]: Advancing to term 4 I0917 09:19:40.802383 14145 raft_consensus.cc:667] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 4 LEADER]: Becoming Leader. State: Replica: 3b4eceaa1f8246de8fd9a3309a08848b, State: Running, Role: LEADER I0917 09:19:40.802534 14145 consensus_queue.cc:206] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e80b3cd6d95403f8499e84cc013b527" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40627 } } peers { permanent_uuid: "3b4eceaa1f8246de8fd9a3309a08848b" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35707 } } I0917 09:19:40.802670 14145 ts_tablet_manager-itest.cc:327] Waiting for servers to agree... I0917 09:19:40.803246 14145 cluster_itest_util.cc:259] Not converged past 4 yet: 4.4 3.3 I0917 09:19:40.904142 14145 cluster_itest_util.cc:259] Not converged past 4 yet: 4.4 3.3 I0917 09:19:41.105096 14145 cluster_itest_util.cc:259] Not converged past 4 yet: 4.4 3.3 I0917 09:19:41.201033 14260 raft_consensus.cc:1149] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 4 LEADER]: Rejecting Update request from peer 7e80b3cd6d95403f8499e84cc013b527 for earlier term 3. Current term is 4. Ops: [] I0917 09:19:41.201300 14383 consensus_queue.cc:995] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b4eceaa1f8246de8fd9a3309a08848b" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35707 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I0917 09:19:41.201385 14383 consensus_queue.cc:1162] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [LEADER]: Peer 3b4eceaa1f8246de8fd9a3309a08848b log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3 I0917 09:19:41.201429 14383 raft_consensus.cc:2885] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 3 LEADER]: Stepping down as leader of term 3 I0917 09:19:41.201436 14383 raft_consensus.cc:705] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7e80b3cd6d95403f8499e84cc013b527, State: Running, Role: LEADER I0917 09:19:41.201489 14383 consensus_queue.cc:229] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e80b3cd6d95403f8499e84cc013b527" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40627 } } peers { permanent_uuid: "3b4eceaa1f8246de8fd9a3309a08848b" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35707 } } I0917 09:19:41.201515 14383 raft_consensus.cc:2890] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 3 FOLLOWER]: Advancing to term 4 I0917 09:19:41.385864 14328 raft_consensus.cc:1184] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 4 FOLLOWER]: Refusing update from remote peer 3b4eceaa1f8246de8fd9a3309a08848b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch) I0917 09:19:41.386191 14385 consensus_queue.cc:984] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e80b3cd6d95403f8499e84cc013b527" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I0917 09:19:41.406116 14145 ts_tablet_manager-itest.cc:324] Electing peer 0... I0917 09:19:41.406152 14145 raft_consensus.cc:405] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 4 LEADER]: Emulating election... I0917 09:19:41.406157 14145 raft_consensus.cc:2885] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 4 LEADER]: Stepping down as leader of term 4 I0917 09:19:41.406163 14145 raft_consensus.cc:705] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3b4eceaa1f8246de8fd9a3309a08848b, State: Running, Role: LEADER I0917 09:19:41.406236 14145 consensus_queue.cc:229] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e80b3cd6d95403f8499e84cc013b527" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40627 } } peers { permanent_uuid: "3b4eceaa1f8246de8fd9a3309a08848b" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35707 } } I0917 09:19:41.406270 14145 raft_consensus.cc:2890] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 4 FOLLOWER]: Advancing to term 5 I0917 09:19:41.406947 14145 raft_consensus.cc:667] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 5 LEADER]: Becoming Leader. State: Replica: 3b4eceaa1f8246de8fd9a3309a08848b, State: Running, Role: LEADER I0917 09:19:41.407022 14145 consensus_queue.cc:206] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e80b3cd6d95403f8499e84cc013b527" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40627 } } peers { permanent_uuid: "3b4eceaa1f8246de8fd9a3309a08848b" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35707 } } I0917 09:19:41.407130 14145 ts_tablet_manager-itest.cc:327] Waiting for servers to agree... I0917 09:19:41.407605 14145 cluster_itest_util.cc:259] Not converged past 5 yet: 5.5 4.4 I0917 09:19:41.508664 14145 cluster_itest_util.cc:259] Not converged past 5 yet: 5.5 4.4 I0917 09:19:41.709668 14145 cluster_itest_util.cc:259] Not converged past 5 yet: 5.5 4.4 I0917 09:19:41.827591 14328 raft_consensus.cc:2890] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 4 FOLLOWER]: Advancing to term 5 I0917 09:19:41.828359 14328 raft_consensus.cc:1184] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 5 FOLLOWER]: Refusing update from remote peer 3b4eceaa1f8246de8fd9a3309a08848b: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch) I0917 09:19:41.828768 14385 consensus_queue.cc:984] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e80b3cd6d95403f8499e84cc013b527" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0917 09:19:42.010787 14145 ts_tablet_manager-itest.cc:324] Electing peer 0... I0917 09:19:42.010824 14145 raft_consensus.cc:405] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 5 LEADER]: Emulating election... I0917 09:19:42.010829 14145 raft_consensus.cc:2885] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 5 LEADER]: Stepping down as leader of term 5 I0917 09:19:42.010835 14145 raft_consensus.cc:705] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3b4eceaa1f8246de8fd9a3309a08848b, State: Running, Role: LEADER I0917 09:19:42.010913 14145 consensus_queue.cc:229] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e80b3cd6d95403f8499e84cc013b527" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40627 } } peers { permanent_uuid: "3b4eceaa1f8246de8fd9a3309a08848b" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35707 } } I0917 09:19:42.010947 14145 raft_consensus.cc:2890] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 5 FOLLOWER]: Advancing to term 6 I0917 09:19:42.011664 14145 raft_consensus.cc:667] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 6 LEADER]: Becoming Leader. State: Replica: 3b4eceaa1f8246de8fd9a3309a08848b, State: Running, Role: LEADER I0917 09:19:42.011740 14145 consensus_queue.cc:206] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e80b3cd6d95403f8499e84cc013b527" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40627 } } peers { permanent_uuid: "3b4eceaa1f8246de8fd9a3309a08848b" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35707 } } I0917 09:19:42.011868 14145 ts_tablet_manager-itest.cc:327] Waiting for servers to agree... I0917 09:19:42.012379 14145 cluster_itest_util.cc:259] Not converged past 6 yet: 6.6 5.5 I0917 09:19:42.113332 14145 cluster_itest_util.cc:259] Not converged past 6 yet: 6.6 5.5 I0917 09:19:42.314450 14145 cluster_itest_util.cc:259] Not converged past 6 yet: 6.6 5.5 I0917 09:19:42.615361 14145 cluster_itest_util.cc:259] Not converged past 6 yet: 6.6 5.5 I0917 09:19:42.635239 14328 raft_consensus.cc:2890] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 5 FOLLOWER]: Advancing to term 6 I0917 09:19:42.635957 14328 raft_consensus.cc:1184] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 6 FOLLOWER]: Refusing update from remote peer 3b4eceaa1f8246de8fd9a3309a08848b: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch) I0917 09:19:42.636217 14389 consensus_queue.cc:984] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e80b3cd6d95403f8499e84cc013b527" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I0917 09:19:43.016299 14145 ts_tablet_manager-itest.cc:324] Electing peer 1... I0917 09:19:43.016342 14145 raft_consensus.cc:405] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 6 FOLLOWER]: Emulating election... I0917 09:19:43.016348 14145 raft_consensus.cc:2890] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 6 FOLLOWER]: Advancing to term 7 I0917 09:19:43.017099 14145 raft_consensus.cc:667] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 7 LEADER]: Becoming Leader. State: Replica: 7e80b3cd6d95403f8499e84cc013b527, State: Running, Role: LEADER I0917 09:19:43.017179 14145 consensus_queue.cc:206] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e80b3cd6d95403f8499e84cc013b527" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40627 } } peers { permanent_uuid: "3b4eceaa1f8246de8fd9a3309a08848b" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35707 } } I0917 09:19:43.017315 14145 ts_tablet_manager-itest.cc:327] Waiting for servers to agree... I0917 09:19:43.017894 14145 cluster_itest_util.cc:259] Not converged past 7 yet: 6.6 7.7 I0917 09:19:43.118782 14145 cluster_itest_util.cc:259] Not converged past 7 yet: 6.6 7.7 I0917 09:19:43.197933 14328 raft_consensus.cc:1149] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 7 LEADER]: Rejecting Update request from peer 3b4eceaa1f8246de8fd9a3309a08848b for earlier term 6. Current term is 7. Ops: [] I0917 09:19:43.198197 14394 consensus_queue.cc:995] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7e80b3cd6d95403f8499e84cc013b527" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40627 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0917 09:19:43.198266 14394 consensus_queue.cc:1162] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [LEADER]: Peer 7e80b3cd6d95403f8499e84cc013b527 log is divergent from this leader: its last log entry 7.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6 I0917 09:19:43.198287 14394 raft_consensus.cc:2885] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 6 LEADER]: Stepping down as leader of term 6 I0917 09:19:43.198297 14394 raft_consensus.cc:705] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3b4eceaa1f8246de8fd9a3309a08848b, State: Running, Role: LEADER I0917 09:19:43.198339 14394 consensus_queue.cc:229] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e80b3cd6d95403f8499e84cc013b527" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40627 } } peers { permanent_uuid: "3b4eceaa1f8246de8fd9a3309a08848b" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35707 } } I0917 09:19:43.198362 14394 raft_consensus.cc:2890] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 6 FOLLOWER]: Advancing to term 7 I0917 09:19:43.319912 14145 cluster_itest_util.cc:259] Not converged past 7 yet: 6.6 7.7 I0917 09:19:43.613386 14260 raft_consensus.cc:1184] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 7 FOLLOWER]: Refusing update from remote peer 7e80b3cd6d95403f8499e84cc013b527: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch) I0917 09:19:43.613683 14396 consensus_queue.cc:984] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b4eceaa1f8246de8fd9a3309a08848b" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0917 09:19:43.620972 14145 ts_tablet_manager-itest.cc:324] Electing peer 1... I0917 09:19:43.621009 14145 raft_consensus.cc:405] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 7 LEADER]: Emulating election... I0917 09:19:43.621016 14145 raft_consensus.cc:2885] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 7 LEADER]: Stepping down as leader of term 7 I0917 09:19:43.621022 14145 raft_consensus.cc:705] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 7e80b3cd6d95403f8499e84cc013b527, State: Running, Role: LEADER I0917 09:19:43.621102 14145 consensus_queue.cc:229] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e80b3cd6d95403f8499e84cc013b527" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40627 } } peers { permanent_uuid: "3b4eceaa1f8246de8fd9a3309a08848b" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35707 } } I0917 09:19:43.621134 14145 raft_consensus.cc:2890] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 7 FOLLOWER]: Advancing to term 8 I0917 09:19:43.621789 14145 raft_consensus.cc:667] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 8 LEADER]: Becoming Leader. State: Replica: 7e80b3cd6d95403f8499e84cc013b527, State: Running, Role: LEADER I0917 09:19:43.621860 14145 consensus_queue.cc:206] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e80b3cd6d95403f8499e84cc013b527" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40627 } } peers { permanent_uuid: "3b4eceaa1f8246de8fd9a3309a08848b" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35707 } } I0917 09:19:43.621966 14145 ts_tablet_manager-itest.cc:327] Waiting for servers to agree... I0917 09:19:43.622606 14145 cluster_itest_util.cc:259] Not converged past 8 yet: 7.7 8.8 I0917 09:19:43.723578 14145 cluster_itest_util.cc:259] Not converged past 8 yet: 7.7 8.8 I0917 09:19:43.924513 14145 cluster_itest_util.cc:259] Not converged past 8 yet: 7.7 8.8 I0917 09:19:44.033831 14260 raft_consensus.cc:2890] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 7 FOLLOWER]: Advancing to term 8 I0917 09:19:44.034576 14260 raft_consensus.cc:1184] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 8 FOLLOWER]: Refusing update from remote peer 7e80b3cd6d95403f8499e84cc013b527: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch) I0917 09:19:44.034873 14397 consensus_queue.cc:984] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b4eceaa1f8246de8fd9a3309a08848b" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I0917 09:19:44.225670 14145 ts_tablet_manager-itest.cc:324] Electing peer 1... I0917 09:19:44.225708 14145 raft_consensus.cc:405] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 8 LEADER]: Emulating election... I0917 09:19:44.225714 14145 raft_consensus.cc:2885] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 8 LEADER]: Stepping down as leader of term 8 I0917 09:19:44.225719 14145 raft_consensus.cc:705] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 7e80b3cd6d95403f8499e84cc013b527, State: Running, Role: LEADER I0917 09:19:44.225800 14145 consensus_queue.cc:229] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e80b3cd6d95403f8499e84cc013b527" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40627 } } peers { permanent_uuid: "3b4eceaa1f8246de8fd9a3309a08848b" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35707 } } I0917 09:19:44.225829 14145 raft_consensus.cc:2890] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 8 FOLLOWER]: Advancing to term 9 I0917 09:19:44.226519 14145 raft_consensus.cc:667] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 9 LEADER]: Becoming Leader. State: Replica: 7e80b3cd6d95403f8499e84cc013b527, State: Running, Role: LEADER I0917 09:19:44.226598 14145 consensus_queue.cc:206] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e80b3cd6d95403f8499e84cc013b527" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40627 } } peers { permanent_uuid: "3b4eceaa1f8246de8fd9a3309a08848b" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35707 } } I0917 09:19:44.226734 14145 ts_tablet_manager-itest.cc:327] Waiting for servers to agree... I0917 09:19:44.227219 14145 cluster_itest_util.cc:259] Not converged past 9 yet: 8.8 9.9 I0917 09:19:44.328138 14145 cluster_itest_util.cc:259] Not converged past 9 yet: 8.8 9.9 I0917 09:19:44.529278 14145 cluster_itest_util.cc:259] Not converged past 9 yet: 8.8 9.9 I0917 09:19:44.616124 14260 raft_consensus.cc:2890] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 8 FOLLOWER]: Advancing to term 9 I0917 09:19:44.616943 14260 raft_consensus.cc:1184] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 9 FOLLOWER]: Refusing update from remote peer 7e80b3cd6d95403f8499e84cc013b527: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch) I0917 09:19:44.617234 14397 consensus_queue.cc:984] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b4eceaa1f8246de8fd9a3309a08848b" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I0917 09:19:44.830258 14145 ts_tablet_manager-itest.cc:324] Electing peer 0... I0917 09:19:44.830294 14145 raft_consensus.cc:405] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 9 FOLLOWER]: Emulating election... I0917 09:19:44.830301 14145 raft_consensus.cc:2890] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 9 FOLLOWER]: Advancing to term 10 I0917 09:19:44.830965 14145 raft_consensus.cc:667] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 10 LEADER]: Becoming Leader. State: Replica: 3b4eceaa1f8246de8fd9a3309a08848b, State: Running, Role: LEADER I0917 09:19:44.831068 14145 consensus_queue.cc:206] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e80b3cd6d95403f8499e84cc013b527" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40627 } } peers { permanent_uuid: "3b4eceaa1f8246de8fd9a3309a08848b" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35707 } } I0917 09:19:44.831182 14145 ts_tablet_manager-itest.cc:327] Waiting for servers to agree... I0917 09:19:44.831883 14145 cluster_itest_util.cc:259] Not converged past 10 yet: 10.10 9.9 I0917 09:19:44.932876 14145 cluster_itest_util.cc:259] Not converged past 10 yet: 10.10 9.9 I0917 09:19:45.133877 14145 cluster_itest_util.cc:259] Not converged past 10 yet: 10.10 9.9 I0917 09:19:45.211560 14260 raft_consensus.cc:1149] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 10 LEADER]: Rejecting Update request from peer 7e80b3cd6d95403f8499e84cc013b527 for earlier term 9. Current term is 10. Ops: [] I0917 09:19:45.211874 14404 consensus_queue.cc:995] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3b4eceaa1f8246de8fd9a3309a08848b" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35707 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s I0917 09:19:45.211954 14404 consensus_queue.cc:1162] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [LEADER]: Peer 3b4eceaa1f8246de8fd9a3309a08848b log is divergent from this leader: its last log entry 10.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9 I0917 09:19:45.211979 14404 raft_consensus.cc:2885] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 9 LEADER]: Stepping down as leader of term 9 I0917 09:19:45.211987 14404 raft_consensus.cc:705] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 7e80b3cd6d95403f8499e84cc013b527, State: Running, Role: LEADER I0917 09:19:45.212049 14404 consensus_queue.cc:229] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e80b3cd6d95403f8499e84cc013b527" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40627 } } peers { permanent_uuid: "3b4eceaa1f8246de8fd9a3309a08848b" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35707 } } I0917 09:19:45.212076 14404 raft_consensus.cc:2890] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 9 FOLLOWER]: Advancing to term 10 I0917 09:19:45.287537 14328 raft_consensus.cc:1184] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 10 FOLLOWER]: Refusing update from remote peer 3b4eceaa1f8246de8fd9a3309a08848b: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch) I0917 09:19:45.288472 14400 consensus_queue.cc:984] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e80b3cd6d95403f8499e84cc013b527" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s I0917 09:19:45.435444 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:45.439126 14145 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0917 09:19:45.439172 14145 tablet_replica.cc:273] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b: stopping tablet replica I0917 09:19:45.439222 14145 raft_consensus.cc:2147] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 10 LEADER]: Raft consensus shutting down. I0917 09:19:45.439277 14145 raft_consensus.cc:2174] T 252b39981f3b42368cd835c1f62536e5 P 3b4eceaa1f8246de8fd9a3309a08848b [term 10 FOLLOWER]: Raft consensus is shut down! I0917 09:19:45.441116 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:45.441143 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:45.441161 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:45.441869 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:45.444459 14145 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0917 09:19:45.444491 14145 tablet_replica.cc:273] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527: stopping tablet replica I0917 09:19:45.444517 14145 raft_consensus.cc:2147] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 10 FOLLOWER]: Raft consensus shutting down. I0917 09:19:45.444536 14145 raft_consensus.cc:2174] T 252b39981f3b42368cd835c1f62536e5 P 7e80b3cd6d95403f8499e84cc013b527 [term 10 FOLLOWER]: Raft consensus is shut down! I0917 09:19:45.456202 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:45.456238 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:45.456264 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:45.457341 14145 master.cc:247] Master@127.13.208.126:42631 shutting down... I0917 09:19:45.459725 14145 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 5bbc635a9c7e42e9af0179b15b50ec45 [term 1 LEADER]: Raft consensus shutting down. I0917 09:19:45.459764 14145 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 5bbc635a9c7e42e9af0179b15b50ec45 [term 1 FOLLOWER]: Raft consensus is shut down! I0917 09:19:45.459822 14145 tablet_replica.cc:273] T 00000000000000000000000000000000 P 5bbc635a9c7e42e9af0179b15b50ec45: stopping tablet replica I0917 09:19:45.471289 14145 master.cc:258] Master@127.13.208.126:42631 shutdown complete. [ OK ] TsTabletManagerITest.TestReportNewLeaderOnLeaderChange (6690 ms) [ RUN ] TsTabletManagerITest.ReportOnReplicaHealthStatus I0917 09:19:45.474895 14145 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.13.208.126:33109 I0917 09:19:45.475809 14145 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 25304us I0917 09:19:45.475860 14145 fs_manager.cc:263] Metadata directory not provided I0917 09:19:45.475875 14145 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0917 09:19:45.475904 14145 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0917 09:19:45.475934 14145 server_base.cc:453] Attempting to create new FS layout instead I0917 09:19:45.476166 14145 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568711978778636-14145/minicluster-data/master-0-root/instance: uuid: "267f2469bb8d4a8dbab505e0e116de32" format_stamp: "Formatted at 2019-09-17 09:19:45 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:45.477210 14145 fs_manager.cc:503] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s I0917 09:19:45.478235 14145 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0917 09:19:45.478826 14145 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:45.478865 14145 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568711978778636-14145/minicluster-data/master-0-root uuid: "267f2469bb8d4a8dbab505e0e116de32" format_stamp: "Formatted at 2019-09-17 09:19:45 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:45.478884 14145 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568711978778636-14145/minicluster-data/master-0-root metadata directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568711978778636-14145/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568711978778636-14145/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0917 09:19:45.499578 14145 rpc_server.cc:205] RPC server started. Bound to: 127.13.208.126:33109 I0917 09:19:45.499763 14145 webserver.cc:374] Webserver started at http://127.13.208.126:46123/ using document root and password file I0917 09:19:45.499987 14467 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0917 09:19:45.500938 14467 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 267f2469bb8d4a8dbab505e0e116de32: Bootstrap starting. I0917 09:19:45.501194 14467 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 267f2469bb8d4a8dbab505e0e116de32: No blocks or log segments found. Creating new log. I0917 09:19:45.501636 14467 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 267f2469bb8d4a8dbab505e0e116de32: No bootstrap required, opened a new log I0917 09:19:45.501817 14467 raft_consensus.cc:340] T 00000000000000000000000000000000 P 267f2469bb8d4a8dbab505e0e116de32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "267f2469bb8d4a8dbab505e0e116de32" member_type: VOTER } I0917 09:19:45.501830 14467 raft_consensus.cc:366] T 00000000000000000000000000000000 P 267f2469bb8d4a8dbab505e0e116de32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0917 09:19:45.501837 14467 raft_consensus.cc:705] T 00000000000000000000000000000000 P 267f2469bb8d4a8dbab505e0e116de32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 267f2469bb8d4a8dbab505e0e116de32, State: Initialized, Role: FOLLOWER I0917 09:19:45.501886 14467 consensus_queue.cc:229] T 00000000000000000000000000000000 P 267f2469bb8d4a8dbab505e0e116de32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "267f2469bb8d4a8dbab505e0e116de32" member_type: VOTER } I0917 09:19:45.501901 14467 raft_consensus.cc:380] T 00000000000000000000000000000000 P 267f2469bb8d4a8dbab505e0e116de32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0917 09:19:45.501907 14467 raft_consensus.cc:472] T 00000000000000000000000000000000 P 267f2469bb8d4a8dbab505e0e116de32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0917 09:19:45.501914 14467 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 267f2469bb8d4a8dbab505e0e116de32 [term 0 FOLLOWER]: Advancing to term 1 I0917 09:19:45.539275 14467 raft_consensus.cc:494] T 00000000000000000000000000000000 P 267f2469bb8d4a8dbab505e0e116de32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "267f2469bb8d4a8dbab505e0e116de32" member_type: VOTER } I0917 09:19:45.539355 14467 leader_election.cc:310] T 00000000000000000000000000000000 P 267f2469bb8d4a8dbab505e0e116de32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 267f2469bb8d4a8dbab505e0e116de32; no voters: I0917 09:19:45.539480 14467 leader_election.cc:296] T 00000000000000000000000000000000 P 267f2469bb8d4a8dbab505e0e116de32 [CANDIDATE]: Term 1 election: Requested vote from peers I0917 09:19:45.539583 14467 sys_catalog.cc:340] T 00000000000000000000000000000000 P 267f2469bb8d4a8dbab505e0e116de32 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "267f2469bb8d4a8dbab505e0e116de32" member_type: VOTER } } I0917 09:19:45.539602 14467 sys_catalog.cc:343] T 00000000000000000000000000000000 P 267f2469bb8d4a8dbab505e0e116de32 [sys.catalog]: This master's current role is: FOLLOWER I0917 09:19:45.539656 14470 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 267f2469bb8d4a8dbab505e0e116de32 [term 1 FOLLOWER]: Leader election won for term 1 I0917 09:19:45.539800 14470 raft_consensus.cc:667] T 00000000000000000000000000000000 P 267f2469bb8d4a8dbab505e0e116de32 [term 1 LEADER]: Becoming Leader. State: Replica: 267f2469bb8d4a8dbab505e0e116de32, State: Running, Role: LEADER I0917 09:19:45.539913 14470 consensus_queue.cc:206] T 00000000000000000000000000000000 P 267f2469bb8d4a8dbab505e0e116de32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "267f2469bb8d4a8dbab505e0e116de32" member_type: VOTER } I0917 09:19:45.540132 14467 sys_catalog.cc:428] T 00000000000000000000000000000000 P 267f2469bb8d4a8dbab505e0e116de32 [sys.catalog]: configured and running, proceeding with master startup. I0917 09:19:45.540228 14145 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0917 09:19:45.540285 14471 sys_catalog.cc:340] T 00000000000000000000000000000000 P 267f2469bb8d4a8dbab505e0e116de32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 267f2469bb8d4a8dbab505e0e116de32. Latest consensus state: current_term: 1 leader_uuid: "267f2469bb8d4a8dbab505e0e116de32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "267f2469bb8d4a8dbab505e0e116de32" member_type: VOTER } } I0917 09:19:45.540324 14471 sys_catalog.cc:343] T 00000000000000000000000000000000 P 267f2469bb8d4a8dbab505e0e116de32 [sys.catalog]: This master's current role is: LEADER I0917 09:19:45.540259 14469 sys_catalog.cc:340] T 00000000000000000000000000000000 P 267f2469bb8d4a8dbab505e0e116de32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "267f2469bb8d4a8dbab505e0e116de32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "267f2469bb8d4a8dbab505e0e116de32" member_type: VOTER } } W0917 09:19:45.540544 14473 catalog_manager.cc:1144] T 00000000000000000000000000000000 P 267f2469bb8d4a8dbab505e0e116de32: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0917 09:19:45.540585 14473 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0917 09:19:45.540549 14469 sys_catalog.cc:343] T 00000000000000000000000000000000 P 267f2469bb8d4a8dbab505e0e116de32 [sys.catalog]: This master's current role is: LEADER I0917 09:19:45.540688 14474 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0917 09:19:45.540843 14474 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0917 09:19:45.555685 14474 catalog_manager.cc:981] Generated new certificate authority record I0917 09:19:45.556228 14474 catalog_manager.cc:1106] Loading token signing keys... I0917 09:19:45.563412 14474 catalog_manager.cc:4435] T 00000000000000000000000000000000 P 267f2469bb8d4a8dbab505e0e116de32: Generated new TSK 0 I0917 09:19:45.573263 14145 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 25304us I0917 09:19:45.573333 14145 fs_manager.cc:263] Metadata directory not provided I0917 09:19:45.573354 14145 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0917 09:19:45.573434 14145 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0917 09:19:45.573467 14145 server_base.cc:453] Attempting to create new FS layout instead I0917 09:19:45.573748 14145 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568711978778636-14145/minicluster-data/ts-0-root/instance: uuid: "179827f0756c461f99525bf240ed8ef1" format_stamp: "Formatted at 2019-09-17 09:19:45 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:45.575078 14145 fs_manager.cc:503] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s I0917 09:19:45.575333 14145 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0917 09:19:45.576103 14145 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:45.576155 14145 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568711978778636-14145/minicluster-data/ts-0-root uuid: "179827f0756c461f99525bf240ed8ef1" format_stamp: "Formatted at 2019-09-17 09:19:45 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:45.576179 14145 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568711978778636-14145/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568711978778636-14145/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568711978778636-14145/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0917 09:19:45.591063 14145 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0917 09:19:45.591089 14145 ts_tablet_manager.cc:403] Registered 0 tablets I0917 09:19:45.593504 14145 rpc_server.cc:205] RPC server started. Bound to: 127.13.208.65:35539 I0917 09:19:45.593696 14145 webserver.cc:374] Webserver started at http://127.13.208.65:41925/ using document root and password file I0917 09:19:45.595198 14145 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 25304us I0917 09:19:45.595268 14145 fs_manager.cc:263] Metadata directory not provided I0917 09:19:45.595309 14145 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0917 09:19:45.595346 14145 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0917 09:19:45.595360 14145 server_base.cc:453] Attempting to create new FS layout instead I0917 09:19:45.595599 14145 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568711978778636-14145/minicluster-data/ts-1-root/instance: uuid: "f44bd2d216ba4e3e931c81a53dd98300" format_stamp: "Formatted at 2019-09-17 09:19:45 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:45.597146 14541 heartbeater.cc:322] Connected to a master server at 127.13.208.126:33109 I0917 09:19:45.597182 14541 heartbeater.cc:406] Registering TS with master... I0917 09:19:45.597190 14145 fs_manager.cc:503] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s I0917 09:19:45.597234 14541 heartbeater.cc:455] Master 127.13.208.126:33109 requested a full tablet report, sending... I0917 09:19:45.597463 14145 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0917 09:19:45.597466 14433 ts_manager.cc:161] Registered new tserver with Master: 179827f0756c461f99525bf240ed8ef1 (127.13.208.65:35539) I0917 09:19:45.598232 14433 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44438 I0917 09:19:45.598474 14145 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:45.598520 14145 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568711978778636-14145/minicluster-data/ts-1-root uuid: "f44bd2d216ba4e3e931c81a53dd98300" format_stamp: "Formatted at 2019-09-17 09:19:45 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:45.598541 14145 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568711978778636-14145/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568711978778636-14145/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568711978778636-14145/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0917 09:19:45.639652 14145 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0917 09:19:45.639677 14145 ts_tablet_manager.cc:403] Registered 0 tablets I0917 09:19:45.641628 14145 rpc_server.cc:205] RPC server started. Bound to: 127.13.208.66:37607 I0917 09:19:45.641855 14145 webserver.cc:374] Webserver started at http://127.13.208.66:45233/ using document root and password file I0917 09:19:45.644593 14145 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 25304us I0917 09:19:45.644675 14145 fs_manager.cc:263] Metadata directory not provided I0917 09:19:45.644738 14145 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0917 09:19:45.644781 14145 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0917 09:19:45.644796 14145 server_base.cc:453] Attempting to create new FS layout instead I0917 09:19:45.645349 14145 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568711978778636-14145/minicluster-data/ts-2-root/instance: uuid: "c1365408e5a541beb0e809a7e0575482" format_stamp: "Formatted at 2019-09-17 09:19:45 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:45.645761 14609 heartbeater.cc:322] Connected to a master server at 127.13.208.126:33109 I0917 09:19:45.645794 14609 heartbeater.cc:406] Registering TS with master... I0917 09:19:45.645853 14609 heartbeater.cc:455] Master 127.13.208.126:33109 requested a full tablet report, sending... I0917 09:19:45.646044 14433 ts_manager.cc:161] Registered new tserver with Master: f44bd2d216ba4e3e931c81a53dd98300 (127.13.208.66:37607) I0917 09:19:45.646482 14433 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44440 I0917 09:19:45.647258 14145 fs_manager.cc:503] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s I0917 09:19:45.647493 14145 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0917 09:19:45.648649 14145 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:45.648690 14145 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568711978778636-14145/minicluster-data/ts-2-root uuid: "c1365408e5a541beb0e809a7e0575482" format_stamp: "Formatted at 2019-09-17 09:19:45 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:45.648715 14145 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568711978778636-14145/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568711978778636-14145/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1568711978778636-14145/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0917 09:19:45.666872 14145 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0917 09:19:45.666900 14145 ts_tablet_manager.cc:403] Registered 0 tablets I0917 09:19:45.670048 14145 rpc_server.cc:205] RPC server started. Bound to: 127.13.208.67:39839 I0917 09:19:45.670344 14145 webserver.cc:374] Webserver started at http://127.13.208.67:34133/ using document root and password file I0917 09:19:45.674054 14676 heartbeater.cc:322] Connected to a master server at 127.13.208.126:33109 I0917 09:19:45.674098 14676 heartbeater.cc:406] Registering TS with master... I0917 09:19:45.674146 14676 heartbeater.cc:455] Master 127.13.208.126:33109 requested a full tablet report, sending... I0917 09:19:45.674381 14433 ts_manager.cc:161] Registered new tserver with Master: c1365408e5a541beb0e809a7e0575482 (127.13.208.67:39839) I0917 09:19:45.674813 14433 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44442 I0917 09:19:45.674918 14145 internal_mini_cluster.cc:328] 3 TS(s) registered with all masters after 0.004454214s I0917 09:19:45.679666 14433 catalog_manager.cc:1440] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44444: name: "test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { } } W0917 09:19:45.679738 14433 catalog_manager.cc:1605] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0917 09:19:45.685871 14508 tablet_service.cc:1022] Processing CreateTablet for tablet 2ad18ea6110347a38a6a797cfb406a1e (table=test-table [id=5a0c8be5b6b94a80a0fce101fb195130]), partition= I0917 09:19:45.685956 14643 tablet_service.cc:1022] Processing CreateTablet for tablet 2ad18ea6110347a38a6a797cfb406a1e (table=test-table [id=5a0c8be5b6b94a80a0fce101fb195130]), partition= I0917 09:19:45.686014 14508 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 2ad18ea6110347a38a6a797cfb406a1e. 1 dirs total, 0 dirs full, 0 dirs failed I0917 09:19:45.686049 14643 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 2ad18ea6110347a38a6a797cfb406a1e. 1 dirs total, 0 dirs full, 0 dirs failed I0917 09:19:45.687320 14691 tablet_bootstrap.cc:439] T 2ad18ea6110347a38a6a797cfb406a1e P 179827f0756c461f99525bf240ed8ef1: Bootstrap starting. I0917 09:19:45.687641 14576 tablet_service.cc:1022] Processing CreateTablet for tablet 2ad18ea6110347a38a6a797cfb406a1e (table=test-table [id=5a0c8be5b6b94a80a0fce101fb195130]), partition= I0917 09:19:45.687722 14576 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 2ad18ea6110347a38a6a797cfb406a1e. 1 dirs total, 0 dirs full, 0 dirs failed I0917 09:19:45.687888 14691 tablet_bootstrap.cc:590] T 2ad18ea6110347a38a6a797cfb406a1e P 179827f0756c461f99525bf240ed8ef1: No blocks or log segments found. Creating new log. I0917 09:19:45.688446 14691 tablet_bootstrap.cc:439] T 2ad18ea6110347a38a6a797cfb406a1e P 179827f0756c461f99525bf240ed8ef1: No bootstrap required, opened a new log I0917 09:19:45.688503 14691 ts_tablet_manager.cc:1085] T 2ad18ea6110347a38a6a797cfb406a1e P 179827f0756c461f99525bf240ed8ef1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I0917 09:19:45.688680 14691 raft_consensus.cc:340] T 2ad18ea6110347a38a6a797cfb406a1e P 179827f0756c461f99525bf240ed8ef1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1365408e5a541beb0e809a7e0575482" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 39839 } } peers { permanent_uuid: "f44bd2d216ba4e3e931c81a53dd98300" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 37607 } } peers { permanent_uuid: "179827f0756c461f99525bf240ed8ef1" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35539 } } I0917 09:19:45.688705 14691 raft_consensus.cc:366] T 2ad18ea6110347a38a6a797cfb406a1e P 179827f0756c461f99525bf240ed8ef1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0917 09:19:45.688715 14691 raft_consensus.cc:705] T 2ad18ea6110347a38a6a797cfb406a1e P 179827f0756c461f99525bf240ed8ef1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 179827f0756c461f99525bf240ed8ef1, State: Initialized, Role: FOLLOWER I0917 09:19:45.688838 14693 tablet_bootstrap.cc:439] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300: Bootstrap starting. I0917 09:19:45.688975 14694 tablet_bootstrap.cc:439] T 2ad18ea6110347a38a6a797cfb406a1e P c1365408e5a541beb0e809a7e0575482: Bootstrap starting. I0917 09:19:45.689175 14694 tablet_bootstrap.cc:590] T 2ad18ea6110347a38a6a797cfb406a1e P c1365408e5a541beb0e809a7e0575482: No blocks or log segments found. Creating new log. I0917 09:19:45.689637 14693 tablet_bootstrap.cc:590] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300: No blocks or log segments found. Creating new log. I0917 09:19:45.689821 14691 consensus_queue.cc:229] T 2ad18ea6110347a38a6a797cfb406a1e P 179827f0756c461f99525bf240ed8ef1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1365408e5a541beb0e809a7e0575482" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 39839 } } peers { permanent_uuid: "f44bd2d216ba4e3e931c81a53dd98300" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 37607 } } peers { permanent_uuid: "179827f0756c461f99525bf240ed8ef1" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35539 } } I0917 09:19:45.689920 14691 ts_tablet_manager.cc:1113] T 2ad18ea6110347a38a6a797cfb406a1e P 179827f0756c461f99525bf240ed8ef1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0917 09:19:45.689947 14541 heartbeater.cc:447] Master 127.13.208.126:33109 was elected leader, sending a full tablet report... I0917 09:19:45.690232 14694 tablet_bootstrap.cc:439] T 2ad18ea6110347a38a6a797cfb406a1e P c1365408e5a541beb0e809a7e0575482: No bootstrap required, opened a new log I0917 09:19:45.690289 14693 tablet_bootstrap.cc:439] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300: No bootstrap required, opened a new log I0917 09:19:45.690337 14693 ts_tablet_manager.cc:1085] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I0917 09:19:45.690338 14694 ts_tablet_manager.cc:1085] T 2ad18ea6110347a38a6a797cfb406a1e P c1365408e5a541beb0e809a7e0575482: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I0917 09:19:45.690510 14693 raft_consensus.cc:340] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1365408e5a541beb0e809a7e0575482" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 39839 } } peers { permanent_uuid: "f44bd2d216ba4e3e931c81a53dd98300" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 37607 } } peers { permanent_uuid: "179827f0756c461f99525bf240ed8ef1" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35539 } } I0917 09:19:45.690524 14693 raft_consensus.cc:366] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0917 09:19:45.690531 14693 raft_consensus.cc:705] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f44bd2d216ba4e3e931c81a53dd98300, State: Initialized, Role: FOLLOWER I0917 09:19:45.690569 14694 raft_consensus.cc:340] T 2ad18ea6110347a38a6a797cfb406a1e P c1365408e5a541beb0e809a7e0575482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1365408e5a541beb0e809a7e0575482" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 39839 } } peers { permanent_uuid: "f44bd2d216ba4e3e931c81a53dd98300" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 37607 } } peers { permanent_uuid: "179827f0756c461f99525bf240ed8ef1" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35539 } } I0917 09:19:45.690587 14694 raft_consensus.cc:366] T 2ad18ea6110347a38a6a797cfb406a1e P c1365408e5a541beb0e809a7e0575482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0917 09:19:45.690595 14694 raft_consensus.cc:705] T 2ad18ea6110347a38a6a797cfb406a1e P c1365408e5a541beb0e809a7e0575482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1365408e5a541beb0e809a7e0575482, State: Initialized, Role: FOLLOWER I0917 09:19:45.690599 14693 consensus_queue.cc:229] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1365408e5a541beb0e809a7e0575482" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 39839 } } peers { permanent_uuid: "f44bd2d216ba4e3e931c81a53dd98300" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 37607 } } peers { permanent_uuid: "179827f0756c461f99525bf240ed8ef1" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35539 } } I0917 09:19:45.690670 14694 consensus_queue.cc:229] T 2ad18ea6110347a38a6a797cfb406a1e P c1365408e5a541beb0e809a7e0575482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1365408e5a541beb0e809a7e0575482" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 39839 } } peers { permanent_uuid: "f44bd2d216ba4e3e931c81a53dd98300" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 37607 } } peers { permanent_uuid: "179827f0756c461f99525bf240ed8ef1" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35539 } } I0917 09:19:45.690702 14693 ts_tablet_manager.cc:1113] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I0917 09:19:45.690729 14609 heartbeater.cc:447] Master 127.13.208.126:33109 was elected leader, sending a full tablet report... I0917 09:19:45.690798 14694 ts_tablet_manager.cc:1113] T 2ad18ea6110347a38a6a797cfb406a1e P c1365408e5a541beb0e809a7e0575482: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I0917 09:19:45.690846 14676 heartbeater.cc:447] Master 127.13.208.126:33109 was elected leader, sending a full tablet report... I0917 09:19:45.717115 14697 raft_consensus.cc:472] T 2ad18ea6110347a38a6a797cfb406a1e P 179827f0756c461f99525bf240ed8ef1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0917 09:19:45.717435 14697 raft_consensus.cc:494] T 2ad18ea6110347a38a6a797cfb406a1e P 179827f0756c461f99525bf240ed8ef1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1365408e5a541beb0e809a7e0575482" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 39839 } } peers { permanent_uuid: "f44bd2d216ba4e3e931c81a53dd98300" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 37607 } } peers { permanent_uuid: "179827f0756c461f99525bf240ed8ef1" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35539 } } I0917 09:19:45.717603 14697 leader_election.cc:296] T 2ad18ea6110347a38a6a797cfb406a1e P 179827f0756c461f99525bf240ed8ef1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1365408e5a541beb0e809a7e0575482 (127.13.208.67:39839), f44bd2d216ba4e3e931c81a53dd98300 (127.13.208.66:37607) I0917 09:19:45.722491 14653 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "2ad18ea6110347a38a6a797cfb406a1e" candidate_uuid: "179827f0756c461f99525bf240ed8ef1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1365408e5a541beb0e809a7e0575482" is_pre_election: true I0917 09:19:45.722504 14586 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "2ad18ea6110347a38a6a797cfb406a1e" candidate_uuid: "179827f0756c461f99525bf240ed8ef1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f44bd2d216ba4e3e931c81a53dd98300" is_pre_election: true I0917 09:19:45.722559 14586 raft_consensus.cc:2350] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 179827f0756c461f99525bf240ed8ef1 in term 0. I0917 09:19:45.722560 14653 raft_consensus.cc:2350] T 2ad18ea6110347a38a6a797cfb406a1e P c1365408e5a541beb0e809a7e0575482 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 179827f0756c461f99525bf240ed8ef1 in term 0. I0917 09:19:45.722662 14481 leader_election.cc:310] T 2ad18ea6110347a38a6a797cfb406a1e P 179827f0756c461f99525bf240ed8ef1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 179827f0756c461f99525bf240ed8ef1, f44bd2d216ba4e3e931c81a53dd98300; no voters: I0917 09:19:45.722735 14697 raft_consensus.cc:2644] T 2ad18ea6110347a38a6a797cfb406a1e P 179827f0756c461f99525bf240ed8ef1 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0917 09:19:45.722762 14697 raft_consensus.cc:472] T 2ad18ea6110347a38a6a797cfb406a1e P 179827f0756c461f99525bf240ed8ef1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0917 09:19:45.722770 14697 raft_consensus.cc:2890] T 2ad18ea6110347a38a6a797cfb406a1e P 179827f0756c461f99525bf240ed8ef1 [term 0 FOLLOWER]: Advancing to term 1 I0917 09:19:45.726172 14697 raft_consensus.cc:494] T 2ad18ea6110347a38a6a797cfb406a1e P 179827f0756c461f99525bf240ed8ef1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1365408e5a541beb0e809a7e0575482" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 39839 } } peers { permanent_uuid: "f44bd2d216ba4e3e931c81a53dd98300" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 37607 } } peers { permanent_uuid: "179827f0756c461f99525bf240ed8ef1" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35539 } } I0917 09:19:45.726435 14697 leader_election.cc:296] T 2ad18ea6110347a38a6a797cfb406a1e P 179827f0756c461f99525bf240ed8ef1 [CANDIDATE]: Term 1 election: Requested vote from peers c1365408e5a541beb0e809a7e0575482 (127.13.208.67:39839), f44bd2d216ba4e3e931c81a53dd98300 (127.13.208.66:37607) I0917 09:19:45.726480 14653 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "2ad18ea6110347a38a6a797cfb406a1e" candidate_uuid: "179827f0756c461f99525bf240ed8ef1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1365408e5a541beb0e809a7e0575482" I0917 09:19:45.726496 14653 raft_consensus.cc:2890] T 2ad18ea6110347a38a6a797cfb406a1e P c1365408e5a541beb0e809a7e0575482 [term 0 FOLLOWER]: Advancing to term 1 I0917 09:19:45.726498 14586 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "2ad18ea6110347a38a6a797cfb406a1e" candidate_uuid: "179827f0756c461f99525bf240ed8ef1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f44bd2d216ba4e3e931c81a53dd98300" I0917 09:19:45.726523 14586 raft_consensus.cc:2890] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300 [term 0 FOLLOWER]: Advancing to term 1 I0917 09:19:45.727037 14653 raft_consensus.cc:2350] T 2ad18ea6110347a38a6a797cfb406a1e P c1365408e5a541beb0e809a7e0575482 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 179827f0756c461f99525bf240ed8ef1 in term 1. I0917 09:19:45.727139 14479 leader_election.cc:310] T 2ad18ea6110347a38a6a797cfb406a1e P 179827f0756c461f99525bf240ed8ef1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 179827f0756c461f99525bf240ed8ef1, c1365408e5a541beb0e809a7e0575482; no voters: I0917 09:19:45.727192 14697 raft_consensus.cc:2644] T 2ad18ea6110347a38a6a797cfb406a1e P 179827f0756c461f99525bf240ed8ef1 [term 1 FOLLOWER]: Leader election won for term 1 I0917 09:19:45.727310 14697 raft_consensus.cc:667] T 2ad18ea6110347a38a6a797cfb406a1e P 179827f0756c461f99525bf240ed8ef1 [term 1 LEADER]: Becoming Leader. State: Replica: 179827f0756c461f99525bf240ed8ef1, State: Running, Role: LEADER I0917 09:19:45.727324 14586 raft_consensus.cc:2350] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 179827f0756c461f99525bf240ed8ef1 in term 1. I0917 09:19:45.727417 14697 consensus_queue.cc:206] T 2ad18ea6110347a38a6a797cfb406a1e P 179827f0756c461f99525bf240ed8ef1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1365408e5a541beb0e809a7e0575482" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 39839 } } peers { permanent_uuid: "f44bd2d216ba4e3e931c81a53dd98300" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 37607 } } peers { permanent_uuid: "179827f0756c461f99525bf240ed8ef1" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35539 } } I0917 09:19:45.727957 14433 catalog_manager.cc:4120] T 2ad18ea6110347a38a6a797cfb406a1e P 179827f0756c461f99525bf240ed8ef1 reported cstate change: term changed from 0 to 1, leader changed from to 179827f0756c461f99525bf240ed8ef1 (127.13.208.65). New cstate: current_term: 1 leader_uuid: "179827f0756c461f99525bf240ed8ef1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1365408e5a541beb0e809a7e0575482" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 39839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f44bd2d216ba4e3e931c81a53dd98300" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 37607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "179827f0756c461f99525bf240ed8ef1" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35539 } health_report { overall_health: HEALTHY } } } I0917 09:19:46.206527 14697 consensus_queue.cc:984] T 2ad18ea6110347a38a6a797cfb406a1e P 179827f0756c461f99525bf240ed8ef1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1365408e5a541beb0e809a7e0575482" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 39839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0917 09:19:46.207762 14701 consensus_queue.cc:984] T 2ad18ea6110347a38a6a797cfb406a1e P 179827f0756c461f99525bf240ed8ef1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f44bd2d216ba4e3e931c81a53dd98300" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 37607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0917 09:19:46.768110 14145 tablet_replica.cc:273] T 2ad18ea6110347a38a6a797cfb406a1e P 179827f0756c461f99525bf240ed8ef1: stopping tablet replica I0917 09:19:46.768173 14145 raft_consensus.cc:2147] T 2ad18ea6110347a38a6a797cfb406a1e P 179827f0756c461f99525bf240ed8ef1 [term 1 LEADER]: Raft consensus shutting down. I0917 09:19:46.768218 14145 raft_consensus.cc:2174] T 2ad18ea6110347a38a6a797cfb406a1e P 179827f0756c461f99525bf240ed8ef1 [term 1 FOLLOWER]: Raft consensus is shut down! I0917 09:19:47.708839 14713 raft_consensus.cc:472] T 2ad18ea6110347a38a6a797cfb406a1e P c1365408e5a541beb0e809a7e0575482 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 179827f0756c461f99525bf240ed8ef1) I0917 09:19:47.709017 14713 raft_consensus.cc:494] T 2ad18ea6110347a38a6a797cfb406a1e P c1365408e5a541beb0e809a7e0575482 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1365408e5a541beb0e809a7e0575482" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 39839 } } peers { permanent_uuid: "f44bd2d216ba4e3e931c81a53dd98300" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 37607 } } peers { permanent_uuid: "179827f0756c461f99525bf240ed8ef1" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35539 } } I0917 09:19:47.709125 14713 leader_election.cc:296] T 2ad18ea6110347a38a6a797cfb406a1e P c1365408e5a541beb0e809a7e0575482 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f44bd2d216ba4e3e931c81a53dd98300 (127.13.208.66:37607), 179827f0756c461f99525bf240ed8ef1 (127.13.208.65:35539) I0917 09:19:47.714277 14586 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "2ad18ea6110347a38a6a797cfb406a1e" candidate_uuid: "c1365408e5a541beb0e809a7e0575482" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f44bd2d216ba4e3e931c81a53dd98300" is_pre_election: true I0917 09:19:47.714339 14586 raft_consensus.cc:2311] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c1365408e5a541beb0e809a7e0575482 for term 2 because replica is either leader or believes a valid leader to be alive. I0917 09:19:47.714450 14518 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "2ad18ea6110347a38a6a797cfb406a1e" candidate_uuid: "c1365408e5a541beb0e809a7e0575482" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "179827f0756c461f99525bf240ed8ef1" is_pre_election: true W0917 09:19:47.714661 14617 leader_election.cc:348] T 2ad18ea6110347a38a6a797cfb406a1e P c1365408e5a541beb0e809a7e0575482 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 179827f0756c461f99525bf240ed8ef1 (127.13.208.65:35539): Illegal state: cannot vote while shut down I0917 09:19:47.714684 14617 leader_election.cc:310] T 2ad18ea6110347a38a6a797cfb406a1e P c1365408e5a541beb0e809a7e0575482 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1365408e5a541beb0e809a7e0575482; no voters: 179827f0756c461f99525bf240ed8ef1, f44bd2d216ba4e3e931c81a53dd98300 I0917 09:19:47.714745 14713 raft_consensus.cc:2592] T 2ad18ea6110347a38a6a797cfb406a1e P c1365408e5a541beb0e809a7e0575482 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0917 09:19:48.225364 14718 raft_consensus.cc:472] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 179827f0756c461f99525bf240ed8ef1) I0917 09:19:48.225484 14718 raft_consensus.cc:494] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1365408e5a541beb0e809a7e0575482" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 39839 } } peers { permanent_uuid: "f44bd2d216ba4e3e931c81a53dd98300" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 37607 } } peers { permanent_uuid: "179827f0756c461f99525bf240ed8ef1" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35539 } } I0917 09:19:48.225662 14718 leader_election.cc:296] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1365408e5a541beb0e809a7e0575482 (127.13.208.67:39839), 179827f0756c461f99525bf240ed8ef1 (127.13.208.65:35539) I0917 09:19:48.229501 14653 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "2ad18ea6110347a38a6a797cfb406a1e" candidate_uuid: "f44bd2d216ba4e3e931c81a53dd98300" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c1365408e5a541beb0e809a7e0575482" is_pre_election: true I0917 09:19:48.229554 14518 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "2ad18ea6110347a38a6a797cfb406a1e" candidate_uuid: "f44bd2d216ba4e3e931c81a53dd98300" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "179827f0756c461f99525bf240ed8ef1" is_pre_election: true I0917 09:19:48.229558 14653 raft_consensus.cc:2350] T 2ad18ea6110347a38a6a797cfb406a1e P c1365408e5a541beb0e809a7e0575482 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f44bd2d216ba4e3e931c81a53dd98300 in term 1. W0917 09:19:48.229662 14550 leader_election.cc:348] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 179827f0756c461f99525bf240ed8ef1 (127.13.208.65:35539): Illegal state: cannot vote while shut down I0917 09:19:48.229693 14547 leader_election.cc:310] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c1365408e5a541beb0e809a7e0575482, f44bd2d216ba4e3e931c81a53dd98300; no voters: 179827f0756c461f99525bf240ed8ef1 I0917 09:19:48.229766 14718 raft_consensus.cc:2644] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0917 09:19:48.229794 14718 raft_consensus.cc:472] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 179827f0756c461f99525bf240ed8ef1) I0917 09:19:48.229802 14718 raft_consensus.cc:2890] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300 [term 1 FOLLOWER]: Advancing to term 2 I0917 09:19:48.230507 14718 raft_consensus.cc:494] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1365408e5a541beb0e809a7e0575482" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 39839 } } peers { permanent_uuid: "f44bd2d216ba4e3e931c81a53dd98300" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 37607 } } peers { permanent_uuid: "179827f0756c461f99525bf240ed8ef1" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35539 } } I0917 09:19:48.230592 14718 leader_election.cc:296] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300 [CANDIDATE]: Term 2 election: Requested vote from peers c1365408e5a541beb0e809a7e0575482 (127.13.208.67:39839), 179827f0756c461f99525bf240ed8ef1 (127.13.208.65:35539) I0917 09:19:48.230846 14653 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "2ad18ea6110347a38a6a797cfb406a1e" candidate_uuid: "f44bd2d216ba4e3e931c81a53dd98300" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c1365408e5a541beb0e809a7e0575482" I0917 09:19:48.230846 14518 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "2ad18ea6110347a38a6a797cfb406a1e" candidate_uuid: "f44bd2d216ba4e3e931c81a53dd98300" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "179827f0756c461f99525bf240ed8ef1" I0917 09:19:48.230895 14653 raft_consensus.cc:2890] T 2ad18ea6110347a38a6a797cfb406a1e P c1365408e5a541beb0e809a7e0575482 [term 1 FOLLOWER]: Advancing to term 2 W0917 09:19:48.231009 14550 leader_election.cc:348] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 179827f0756c461f99525bf240ed8ef1 (127.13.208.65:35539): Illegal state: cannot vote while shut down I0917 09:19:48.231484 14653 raft_consensus.cc:2350] T 2ad18ea6110347a38a6a797cfb406a1e P c1365408e5a541beb0e809a7e0575482 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f44bd2d216ba4e3e931c81a53dd98300 in term 2. I0917 09:19:48.231634 14547 leader_election.cc:310] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c1365408e5a541beb0e809a7e0575482, f44bd2d216ba4e3e931c81a53dd98300; no voters: 179827f0756c461f99525bf240ed8ef1 I0917 09:19:48.231757 14718 raft_consensus.cc:2644] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300 [term 2 FOLLOWER]: Leader election won for term 2 I0917 09:19:48.231881 14718 raft_consensus.cc:667] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300 [term 2 LEADER]: Becoming Leader. State: Replica: f44bd2d216ba4e3e931c81a53dd98300, State: Running, Role: LEADER I0917 09:19:48.231973 14718 consensus_queue.cc:206] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1365408e5a541beb0e809a7e0575482" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 39839 } } peers { permanent_uuid: "f44bd2d216ba4e3e931c81a53dd98300" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 37607 } } peers { permanent_uuid: "179827f0756c461f99525bf240ed8ef1" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 35539 } } I0917 09:19:48.631811 14653 raft_consensus.cc:1184] T 2ad18ea6110347a38a6a797cfb406a1e P c1365408e5a541beb0e809a7e0575482 [term 2 FOLLOWER]: Refusing update from remote peer f44bd2d216ba4e3e931c81a53dd98300: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch) I0917 09:19:48.632179 14718 consensus_queue.cc:984] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1365408e5a541beb0e809a7e0575482" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 39839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0917 09:19:48.633466 14550 consensus_queue.cc:544] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300 [LEADER]: The tablet replica hosted on peer 179827f0756c461f99525bf240ed8ef1 has failed W0917 09:19:48.633563 14550 consensus_peers.cc:458] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300 -> Peer 179827f0756c461f99525bf240ed8ef1 (127.13.208.65:35539): Couldn't send request to peer 179827f0756c461f99525bf240ed8ef1. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry. I0917 09:19:48.794059 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:48.797482 14145 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0917 09:19:48.809214 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:48.809252 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:48.809278 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:48.809919 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:48.812758 14145 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0917 09:19:48.812795 14145 tablet_replica.cc:273] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300: stopping tablet replica I0917 09:19:48.812825 14145 raft_consensus.cc:2147] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300 [term 2 LEADER]: Raft consensus shutting down. I0917 09:19:48.812916 14145 raft_consensus.cc:2174] T 2ad18ea6110347a38a6a797cfb406a1e P f44bd2d216ba4e3e931c81a53dd98300 [term 2 FOLLOWER]: Raft consensus is shut down! I0917 09:19:48.825026 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:48.825060 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:48.825085 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:48.825887 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:48.828766 14145 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0917 09:19:48.828809 14145 tablet_replica.cc:273] T 2ad18ea6110347a38a6a797cfb406a1e P c1365408e5a541beb0e809a7e0575482: stopping tablet replica I0917 09:19:48.828850 14145 raft_consensus.cc:2147] T 2ad18ea6110347a38a6a797cfb406a1e P c1365408e5a541beb0e809a7e0575482 [term 2 FOLLOWER]: Raft consensus shutting down. I0917 09:19:48.828877 14145 raft_consensus.cc:2174] T 2ad18ea6110347a38a6a797cfb406a1e P c1365408e5a541beb0e809a7e0575482 [term 2 FOLLOWER]: Raft consensus is shut down! I0917 09:19:48.840711 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:48.840749 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:48.840772 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:48.841526 14145 master.cc:247] Master@127.13.208.126:33109 shutting down... I0917 09:19:48.843703 14145 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 267f2469bb8d4a8dbab505e0e116de32 [term 1 LEADER]: Raft consensus shutting down. I0917 09:19:48.843737 14145 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 267f2469bb8d4a8dbab505e0e116de32 [term 1 FOLLOWER]: Raft consensus is shut down! I0917 09:19:48.843750 14145 tablet_replica.cc:273] T 00000000000000000000000000000000 P 267f2469bb8d4a8dbab505e0e116de32: stopping tablet replica I0917 09:19:48.855087 14145 master.cc:258] Master@127.13.208.126:33109 shutdown complete. [ OK ] TsTabletManagerITest.ReportOnReplicaHealthStatus (3384 ms) [ RUN ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters I0917 09:19:48.858986 14145 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.13.208.126:39773 I0917 09:19:48.859807 14145 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 26804us I0917 09:19:48.859858 14145 fs_manager.cc:263] Metadata directory not provided I0917 09:19:48.859882 14145 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0917 09:19:48.859915 14145 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0917 09:19:48.859941 14145 server_base.cc:453] Attempting to create new FS layout instead I0917 09:19:48.860177 14145 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568711978778636-14145/minicluster-data/master-0-root/instance: uuid: "caf14a87cced475aba5bd0ed05a0d6e7" format_stamp: "Formatted at 2019-09-17 09:19:48 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:48.861281 14145 fs_manager.cc:503] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s I0917 09:19:48.861560 14145 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0917 09:19:48.862255 14145 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:48.862300 14145 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568711978778636-14145/minicluster-data/master-0-root uuid: "caf14a87cced475aba5bd0ed05a0d6e7" format_stamp: "Formatted at 2019-09-17 09:19:48 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:48.862356 14145 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568711978778636-14145/minicluster-data/master-0-root metadata directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568711978778636-14145/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568711978778636-14145/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0917 09:19:48.869989 14145 rpc_server.cc:205] RPC server started. Bound to: 127.13.208.126:39773 I0917 09:19:48.870193 14145 webserver.cc:374] Webserver started at http://127.13.208.126:35363/ using document root and password file I0917 09:19:48.870498 14788 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0917 09:19:48.871438 14788 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P caf14a87cced475aba5bd0ed05a0d6e7: Bootstrap starting. I0917 09:19:48.871688 14788 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P caf14a87cced475aba5bd0ed05a0d6e7: No blocks or log segments found. Creating new log. I0917 09:19:48.872155 14788 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P caf14a87cced475aba5bd0ed05a0d6e7: No bootstrap required, opened a new log I0917 09:19:48.872299 14788 raft_consensus.cc:340] T 00000000000000000000000000000000 P caf14a87cced475aba5bd0ed05a0d6e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf14a87cced475aba5bd0ed05a0d6e7" member_type: VOTER } I0917 09:19:48.872318 14788 raft_consensus.cc:366] T 00000000000000000000000000000000 P caf14a87cced475aba5bd0ed05a0d6e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0917 09:19:48.872324 14788 raft_consensus.cc:705] T 00000000000000000000000000000000 P caf14a87cced475aba5bd0ed05a0d6e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caf14a87cced475aba5bd0ed05a0d6e7, State: Initialized, Role: FOLLOWER I0917 09:19:48.872365 14788 consensus_queue.cc:229] T 00000000000000000000000000000000 P caf14a87cced475aba5bd0ed05a0d6e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf14a87cced475aba5bd0ed05a0d6e7" member_type: VOTER } I0917 09:19:48.872373 14788 raft_consensus.cc:380] T 00000000000000000000000000000000 P caf14a87cced475aba5bd0ed05a0d6e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0917 09:19:48.872380 14788 raft_consensus.cc:472] T 00000000000000000000000000000000 P caf14a87cced475aba5bd0ed05a0d6e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0917 09:19:48.872387 14788 raft_consensus.cc:2890] T 00000000000000000000000000000000 P caf14a87cced475aba5bd0ed05a0d6e7 [term 0 FOLLOWER]: Advancing to term 1 I0917 09:19:48.872964 14788 raft_consensus.cc:494] T 00000000000000000000000000000000 P caf14a87cced475aba5bd0ed05a0d6e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf14a87cced475aba5bd0ed05a0d6e7" member_type: VOTER } I0917 09:19:48.872995 14788 leader_election.cc:310] T 00000000000000000000000000000000 P caf14a87cced475aba5bd0ed05a0d6e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: caf14a87cced475aba5bd0ed05a0d6e7; no voters: I0917 09:19:48.873081 14788 leader_election.cc:296] T 00000000000000000000000000000000 P caf14a87cced475aba5bd0ed05a0d6e7 [CANDIDATE]: Term 1 election: Requested vote from peers I0917 09:19:48.873160 14790 raft_consensus.cc:2644] T 00000000000000000000000000000000 P caf14a87cced475aba5bd0ed05a0d6e7 [term 1 FOLLOWER]: Leader election won for term 1 I0917 09:19:48.873260 14790 raft_consensus.cc:667] T 00000000000000000000000000000000 P caf14a87cced475aba5bd0ed05a0d6e7 [term 1 LEADER]: Becoming Leader. State: Replica: caf14a87cced475aba5bd0ed05a0d6e7, State: Running, Role: LEADER I0917 09:19:48.873317 14790 consensus_queue.cc:206] T 00000000000000000000000000000000 P caf14a87cced475aba5bd0ed05a0d6e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf14a87cced475aba5bd0ed05a0d6e7" member_type: VOTER } I0917 09:19:48.873445 14790 sys_catalog.cc:340] T 00000000000000000000000000000000 P caf14a87cced475aba5bd0ed05a0d6e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "caf14a87cced475aba5bd0ed05a0d6e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf14a87cced475aba5bd0ed05a0d6e7" member_type: VOTER } } I0917 09:19:48.873458 14790 sys_catalog.cc:343] T 00000000000000000000000000000000 P caf14a87cced475aba5bd0ed05a0d6e7 [sys.catalog]: This master's current role is: LEADER I0917 09:19:48.873515 14788 sys_catalog.cc:340] T 00000000000000000000000000000000 P caf14a87cced475aba5bd0ed05a0d6e7 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "caf14a87cced475aba5bd0ed05a0d6e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf14a87cced475aba5bd0ed05a0d6e7" member_type: VOTER } } I0917 09:19:48.873548 14788 sys_catalog.cc:343] T 00000000000000000000000000000000 P caf14a87cced475aba5bd0ed05a0d6e7 [sys.catalog]: This master's current role is: LEADER I0917 09:19:48.873623 14788 sys_catalog.cc:428] T 00000000000000000000000000000000 P caf14a87cced475aba5bd0ed05a0d6e7 [sys.catalog]: configured and running, proceeding with master startup. I0917 09:19:48.873708 14790 sys_catalog.cc:340] T 00000000000000000000000000000000 P caf14a87cced475aba5bd0ed05a0d6e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader caf14a87cced475aba5bd0ed05a0d6e7. Latest consensus state: current_term: 1 leader_uuid: "caf14a87cced475aba5bd0ed05a0d6e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf14a87cced475aba5bd0ed05a0d6e7" member_type: VOTER } } I0917 09:19:48.873721 14790 sys_catalog.cc:343] T 00000000000000000000000000000000 P caf14a87cced475aba5bd0ed05a0d6e7 [sys.catalog]: This master's current role is: LEADER I0917 09:19:48.873842 14794 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0917 09:19:48.873911 14145 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0917 09:19:48.874037 14794 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0917 09:19:48.888208 14794 catalog_manager.cc:981] Generated new certificate authority record I0917 09:19:48.888715 14794 catalog_manager.cc:1106] Loading token signing keys... I0917 09:19:48.896029 14794 catalog_manager.cc:4435] T 00000000000000000000000000000000 P caf14a87cced475aba5bd0ed05a0d6e7: Generated new TSK 0 I0917 09:19:48.906677 14145 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 26804us I0917 09:19:48.906750 14145 fs_manager.cc:263] Metadata directory not provided I0917 09:19:48.906770 14145 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0917 09:19:48.906798 14145 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0917 09:19:48.906803 14145 server_base.cc:453] Attempting to create new FS layout instead I0917 09:19:48.907061 14145 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568711978778636-14145/minicluster-data/ts-0-root/instance: uuid: "daa861c299924377baebdf267393ab99" format_stamp: "Formatted at 2019-09-17 09:19:48 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:48.908344 14145 fs_manager.cc:503] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s I0917 09:19:48.908591 14145 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0917 09:19:48.909307 14145 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:48.909348 14145 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568711978778636-14145/minicluster-data/ts-0-root uuid: "daa861c299924377baebdf267393ab99" format_stamp: "Formatted at 2019-09-17 09:19:48 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:48.909366 14145 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568711978778636-14145/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568711978778636-14145/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568711978778636-14145/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0917 09:19:48.939936 14145 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0917 09:19:48.939960 14145 ts_tablet_manager.cc:403] Registered 0 tablets I0917 09:19:48.942270 14145 rpc_server.cc:205] RPC server started. Bound to: 127.13.208.65:43185 I0917 09:19:48.942510 14145 webserver.cc:374] Webserver started at http://127.13.208.65:33867/ using document root and password file I0917 09:19:48.945976 14862 heartbeater.cc:322] Connected to a master server at 127.13.208.126:39773 I0917 09:19:48.946018 14862 heartbeater.cc:406] Registering TS with master... I0917 09:19:48.946069 14862 heartbeater.cc:455] Master 127.13.208.126:39773 requested a full tablet report, sending... I0917 09:19:48.946311 14755 ts_manager.cc:161] Registered new tserver with Master: daa861c299924377baebdf267393ab99 (127.13.208.65:43185) I0917 09:19:48.946911 14755 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42524 I0917 09:19:48.947021 14145 internal_mini_cluster.cc:328] 1 TS(s) registered with all masters after 0.004381748s I0917 09:19:48.951117 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:48.954124 14145 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0917 09:19:48.965319 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:48.965351 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:48.965375 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:48.966966 14145 tablet_server.cc:87] deduplicated master addresses: 127.13.208.126:39773 I0917 09:19:48.967012 14145 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 26804us I0917 09:19:48.967063 14145 fs_manager.cc:263] Metadata directory not provided I0917 09:19:48.967087 14145 fs_manager.cc:266] Using existing metadata directory in first data directory I0917 09:19:48.967538 14145 fs_manager.cc:399] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:48.967730 14145 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0917 09:19:48.968360 14145 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:48.968421 14145 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568711978778636-14145/minicluster-data/ts-0-root uuid: "daa861c299924377baebdf267393ab99" format_stamp: "Formatted at 2019-09-17 09:19:48 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:48.968436 14145 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568711978778636-14145/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568711978778636-14145/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1568711978778636-14145/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0917 09:19:48.976040 14145 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0917 09:19:48.976061 14145 ts_tablet_manager.cc:403] Registered 0 tablets I0917 09:19:48.978389 14145 rpc_server.cc:205] RPC server started. Bound to: 127.13.208.65:43185 I0917 09:19:48.978629 14145 webserver.cc:374] Webserver started at http://127.13.208.65:33867/ using document root and password file I0917 09:19:48.979387 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:48.983505 14936 heartbeater.cc:322] Connected to a master server at 127.13.208.126:39773 I0917 09:19:48.983568 14936 heartbeater.cc:406] Registering TS with master... I0917 09:19:48.983634 14936 heartbeater.cc:455] Master 127.13.208.126:39773 requested a full tablet report, sending... I0917 09:19:48.983949 14755 ts_manager.cc:161] Re-registered known tserver with Master: daa861c299924377baebdf267393ab99 (127.13.208.65:43185) I0917 09:19:48.984427 14755 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42528 I0917 09:19:48.984944 14145 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0917 09:19:48.996150 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:48.996187 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:48.996210 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:48.996863 14145 master.cc:247] Master@127.13.208.126:39773 shutting down... I0917 09:19:49.000097 14145 raft_consensus.cc:2147] T 00000000000000000000000000000000 P caf14a87cced475aba5bd0ed05a0d6e7 [term 1 LEADER]: Raft consensus shutting down. I0917 09:19:49.000133 14145 raft_consensus.cc:2174] T 00000000000000000000000000000000 P caf14a87cced475aba5bd0ed05a0d6e7 [term 1 FOLLOWER]: Raft consensus is shut down! I0917 09:19:49.000190 14145 tablet_replica.cc:273] T 00000000000000000000000000000000 P caf14a87cced475aba5bd0ed05a0d6e7: stopping tablet replica I0917 09:19:49.002028 14145 master.cc:258] Master@127.13.208.126:39773 shutdown complete. [ OK ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters (146 ms) [ RUN ] TsTabletManagerITest.TestTableStats I0917 09:19:49.004766 14145 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.13.208.126:33685,127.13.208.125:37503,127.13.208.124:39307 I0917 09:19:49.005621 14145 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 27304us I0917 09:19:49.005672 14145 fs_manager.cc:263] Metadata directory not provided I0917 09:19:49.005687 14145 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0917 09:19:49.005710 14145 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0917 09:19:49.005719 14145 server_base.cc:453] Attempting to create new FS layout instead I0917 09:19:49.005924 14145 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/master-0-root/instance: uuid: "abfd3dc05c4a4566bb328f04255eb266" format_stamp: "Formatted at 2019-09-17 09:19:49 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:49.006923 14145 fs_manager.cc:503] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s I0917 09:19:49.007151 14145 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0917 09:19:49.007688 14145 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:49.007720 14145 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/master-0-root uuid: "abfd3dc05c4a4566bb328f04255eb266" format_stamp: "Formatted at 2019-09-17 09:19:49 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:49.007740 14145 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/master-0-root metadata directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0917 09:19:49.042817 14145 rpc_server.cc:205] RPC server started. Bound to: 127.13.208.126:33685 I0917 09:19:49.043027 14145 webserver.cc:374] Webserver started at http://127.13.208.126:44757/ using document root and password file I0917 09:19:49.043380 14997 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0917 09:19:49.044049 14997 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } has no permanent_uuid. Determining permanent_uuid... I0917 09:19:49.044272 14145 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 27304us I0917 09:19:49.044330 14145 fs_manager.cc:263] Metadata directory not provided I0917 09:19:49.044359 14145 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0917 09:19:49.044394 14145 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0917 09:19:49.044400 14145 server_base.cc:453] Attempting to create new FS layout instead I0917 09:19:49.044600 14145 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/master-1-root/instance: uuid: "82e2633778254c44a45393ac32540f27" format_stamp: "Formatted at 2019-09-17 09:19:49 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:49.046161 14145 fs_manager.cc:503] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s I0917 09:19:49.046407 14145 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0917 09:19:49.047034 14145 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:49.047066 14145 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/master-1-root uuid: "82e2633778254c44a45393ac32540f27" format_stamp: "Formatted at 2019-09-17 09:19:49 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:49.047087 14145 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/master-1-root metadata directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/master-1-root 1 data directories: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/master-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0917 09:19:49.048038 14997 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } has no permanent_uuid. Determining permanent_uuid... W0917 09:19:49.048298 14997 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.13.208.125:37503: Network error: Client connection negotiation failed: client connection to 127.13.208.125:37503: connect: Connection refused (error 111) I0917 09:19:49.076517 14997 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } attempt: 1 W0917 09:19:49.077006 14997 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.13.208.125:37503: Network error: Client connection negotiation failed: client connection to 127.13.208.125:37503: connect: Connection refused (error 111) I0917 09:19:49.081262 14145 rpc_server.cc:205] RPC server started. Bound to: 127.13.208.125:37503 I0917 09:19:49.081475 14145 webserver.cc:374] Webserver started at http://127.13.208.125:42235/ using document root and password file I0917 09:19:49.081719 15054 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0917 09:19:49.082406 15054 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } has no permanent_uuid. Determining permanent_uuid... I0917 09:19:49.082769 14145 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 27304us I0917 09:19:49.082828 14145 fs_manager.cc:263] Metadata directory not provided I0917 09:19:49.082855 14145 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0917 09:19:49.082906 14145 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0917 09:19:49.082917 14145 server_base.cc:453] Attempting to create new FS layout instead I0917 09:19:49.083130 14145 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/master-2-root/instance: uuid: "e501d8ef778246b69b34d86e7395022b" format_stamp: "Formatted at 2019-09-17 09:19:49 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:49.084308 14145 fs_manager.cc:503] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s I0917 09:19:49.084542 14145 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0917 09:19:49.085347 14145 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:49.085386 14145 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/master-2-root uuid: "e501d8ef778246b69b34d86e7395022b" format_stamp: "Formatted at 2019-09-17 09:19:49 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:49.085427 14145 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/master-2-root metadata directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/master-2-root 1 data directories: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/master-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0917 09:19:49.085498 15054 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } has no permanent_uuid. Determining permanent_uuid... I0917 09:19:49.088338 15054 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } has no permanent_uuid. Determining permanent_uuid... W0917 09:19:49.088632 15054 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.13.208.124:39307: Network error: Client connection negotiation failed: client connection to 127.13.208.124:39307: connect: Connection refused (error 111) I0917 09:19:49.101516 14145 rpc_server.cc:205] RPC server started. Bound to: 127.13.208.124:39307 I0917 09:19:49.101737 14145 webserver.cc:374] Webserver started at http://127.13.208.124:44755/ using document root and password file I0917 09:19:49.101805 14145 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0917 09:19:49.102032 15111 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0917 09:19:49.102692 15111 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } has no permanent_uuid. Determining permanent_uuid... I0917 09:19:49.105717 15111 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } has no permanent_uuid. Determining permanent_uuid... I0917 09:19:49.108317 15111 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } has no permanent_uuid. Determining permanent_uuid... I0917 09:19:49.111649 15111 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b: Bootstrap starting. I0917 09:19:49.111909 15111 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b: No blocks or log segments found. Creating new log. I0917 09:19:49.112336 15111 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b: No bootstrap required, opened a new log I0917 09:19:49.112478 15111 raft_consensus.cc:340] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } I0917 09:19:49.112495 15111 raft_consensus.cc:366] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0917 09:19:49.112500 15111 raft_consensus.cc:705] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e501d8ef778246b69b34d86e7395022b, State: Initialized, Role: FOLLOWER I0917 09:19:49.112545 15111 consensus_queue.cc:229] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } I0917 09:19:49.112686 15111 sys_catalog.cc:340] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:49.112707 15111 sys_catalog.cc:343] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [sys.catalog]: This master's current role is: FOLLOWER I0917 09:19:49.112792 15115 sys_catalog.cc:340] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:49.112808 15111 sys_catalog.cc:428] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [sys.catalog]: configured and running, proceeding with master startup. I0917 09:19:49.112814 15115 sys_catalog.cc:343] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [sys.catalog]: This master's current role is: FOLLOWER W0917 09:19:49.113049 15116 catalog_manager.cc:1144] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0917 09:19:49.113063 15116 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0917 09:19:49.114145 14997 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } attempt: 2 I0917 09:19:49.117017 14997 sys_catalog.cc:307] member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } has no permanent_uuid. Determining permanent_uuid... I0917 09:19:49.120115 14997 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266: Bootstrap starting. I0917 09:19:49.120373 14997 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266: No blocks or log segments found. Creating new log. I0917 09:19:49.120823 14997 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266: No bootstrap required, opened a new log I0917 09:19:49.121052 14997 raft_consensus.cc:340] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } I0917 09:19:49.121068 14997 raft_consensus.cc:366] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0917 09:19:49.121076 14997 raft_consensus.cc:705] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abfd3dc05c4a4566bb328f04255eb266, State: Initialized, Role: FOLLOWER I0917 09:19:49.121152 14997 consensus_queue.cc:229] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } I0917 09:19:49.121316 14997 sys_catalog.cc:340] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:49.121340 15118 sys_catalog.cc:340] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:49.121346 14997 sys_catalog.cc:343] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [sys.catalog]: This master's current role is: FOLLOWER I0917 09:19:49.121358 15118 sys_catalog.cc:343] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [sys.catalog]: This master's current role is: FOLLOWER I0917 09:19:49.121410 14997 sys_catalog.cc:428] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [sys.catalog]: configured and running, proceeding with master startup. I0917 09:19:49.121495 14145 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 1 W0917 09:19:49.121723 15119 catalog_manager.cc:1144] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0917 09:19:49.121738 15119 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0917 09:19:49.145865 15054 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } attempt: 1 I0917 09:19:49.149585 15054 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27: Bootstrap starting. I0917 09:19:49.149863 15054 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27: No blocks or log segments found. Creating new log. I0917 09:19:49.150329 15054 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27: No bootstrap required, opened a new log I0917 09:19:49.150492 15054 raft_consensus.cc:340] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } I0917 09:19:49.150508 15054 raft_consensus.cc:366] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0917 09:19:49.150514 15054 raft_consensus.cc:705] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82e2633778254c44a45393ac32540f27, State: Initialized, Role: FOLLOWER I0917 09:19:49.150571 15054 consensus_queue.cc:229] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } I0917 09:19:49.150710 15054 sys_catalog.cc:340] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:49.150732 15054 sys_catalog.cc:343] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [sys.catalog]: This master's current role is: FOLLOWER I0917 09:19:49.150811 15121 sys_catalog.cc:340] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:49.150832 15121 sys_catalog.cc:343] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [sys.catalog]: This master's current role is: FOLLOWER I0917 09:19:49.150842 15054 sys_catalog.cc:428] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [sys.catalog]: configured and running, proceeding with master startup. I0917 09:19:49.150930 14145 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 2 W0917 09:19:49.151185 15122 catalog_manager.cc:1144] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0917 09:19:49.151204 15122 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0917 09:19:49.152916 14145 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 27304us I0917 09:19:49.152981 14145 fs_manager.cc:263] Metadata directory not provided I0917 09:19:49.153010 14145 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0917 09:19:49.153048 14145 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0917 09:19:49.153059 14145 server_base.cc:453] Attempting to create new FS layout instead I0917 09:19:49.153293 14145 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/ts-0-root/instance: uuid: "c2c8bec4d40141a6b1f470f6142de67a" format_stamp: "Formatted at 2019-09-17 09:19:49 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:49.154816 14145 fs_manager.cc:503] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s I0917 09:19:49.155067 14145 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0917 09:19:49.156077 14145 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:49.156118 14145 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/ts-0-root uuid: "c2c8bec4d40141a6b1f470f6142de67a" format_stamp: "Formatted at 2019-09-17 09:19:49 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:49.156147 14145 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0917 09:19:49.175468 15118 raft_consensus.cc:472] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0917 09:19:49.175606 15118 raft_consensus.cc:494] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } I0917 09:19:49.175911 15115 raft_consensus.cc:472] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0917 09:19:49.175927 15118 leader_election.cc:296] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82e2633778254c44a45393ac32540f27 (127.13.208.125:37503), e501d8ef778246b69b34d86e7395022b (127.13.208.124:39307) I0917 09:19:49.176009 15088 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "abfd3dc05c4a4566bb328f04255eb266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e501d8ef778246b69b34d86e7395022b" is_pre_election: true I0917 09:19:49.176066 15115 raft_consensus.cc:494] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } I0917 09:19:49.176096 15088 raft_consensus.cc:2350] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abfd3dc05c4a4566bb328f04255eb266 in term 0. I0917 09:19:49.176136 15115 leader_election.cc:296] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abfd3dc05c4a4566bb328f04255eb266 (127.13.208.126:33685), 82e2633778254c44a45393ac32540f27 (127.13.208.125:37503) I0917 09:19:49.176190 15031 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "abfd3dc05c4a4566bb328f04255eb266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82e2633778254c44a45393ac32540f27" is_pre_election: true I0917 09:19:49.176215 15031 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abfd3dc05c4a4566bb328f04255eb266 in term 0. I0917 09:19:49.176331 14945 leader_election.cc:310] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82e2633778254c44a45393ac32540f27, abfd3dc05c4a4566bb328f04255eb266; no voters: I0917 09:19:49.176380 14965 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e501d8ef778246b69b34d86e7395022b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abfd3dc05c4a4566bb328f04255eb266" is_pre_election: true I0917 09:19:49.176401 14965 raft_consensus.cc:2350] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e501d8ef778246b69b34d86e7395022b in term 0. I0917 09:19:49.176479 15118 raft_consensus.cc:2644] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0917 09:19:49.176501 15059 leader_election.cc:310] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abfd3dc05c4a4566bb328f04255eb266, e501d8ef778246b69b34d86e7395022b; no voters: I0917 09:19:49.176569 15031 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e501d8ef778246b69b34d86e7395022b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82e2633778254c44a45393ac32540f27" is_pre_election: true I0917 09:19:49.176594 15031 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e501d8ef778246b69b34d86e7395022b in term 0. I0917 09:19:49.176615 15115 raft_consensus.cc:2644] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 0 FOLLOWER]: Leader pre-election won for term 1 I0917 09:19:49.176627 15115 raft_consensus.cc:472] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0917 09:19:49.176635 15115 raft_consensus.cc:2890] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 0 FOLLOWER]: Advancing to term 1 I0917 09:19:49.176695 15118 raft_consensus.cc:472] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0917 09:19:49.176708 15118 raft_consensus.cc:2890] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 0 FOLLOWER]: Advancing to term 1 I0917 09:19:49.178072 15118 raft_consensus.cc:494] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } I0917 09:19:49.178202 15115 raft_consensus.cc:494] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } I0917 09:19:49.178294 15115 leader_election.cc:296] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [CANDIDATE]: Term 1 election: Requested vote from peers abfd3dc05c4a4566bb328f04255eb266 (127.13.208.126:33685), 82e2633778254c44a45393ac32540f27 (127.13.208.125:37503) I0917 09:19:49.178321 15118 leader_election.cc:296] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [CANDIDATE]: Term 1 election: Requested vote from peers 82e2633778254c44a45393ac32540f27 (127.13.208.125:37503), e501d8ef778246b69b34d86e7395022b (127.13.208.124:39307) I0917 09:19:49.178349 15031 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "abfd3dc05c4a4566bb328f04255eb266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82e2633778254c44a45393ac32540f27" I0917 09:19:49.178370 15031 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 0 FOLLOWER]: Advancing to term 1 I0917 09:19:49.178602 15088 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "abfd3dc05c4a4566bb328f04255eb266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e501d8ef778246b69b34d86e7395022b" I0917 09:19:49.178673 14965 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e501d8ef778246b69b34d86e7395022b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abfd3dc05c4a4566bb328f04255eb266" I0917 09:19:49.178753 14965 raft_consensus.cc:2280] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e501d8ef778246b69b34d86e7395022b in current term 1: Already voted for candidate abfd3dc05c4a4566bb328f04255eb266 in this term. I0917 09:19:49.178813 15030 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e501d8ef778246b69b34d86e7395022b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82e2633778254c44a45393ac32540f27" I0917 09:19:49.178721 15088 raft_consensus.cc:2280] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate abfd3dc05c4a4566bb328f04255eb266 in current term 1: Already voted for candidate e501d8ef778246b69b34d86e7395022b in this term. I0917 09:19:49.179368 15031 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abfd3dc05c4a4566bb328f04255eb266 in term 1. I0917 09:19:49.179421 15030 raft_consensus.cc:2325] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e501d8ef778246b69b34d86e7395022b for term 1 because replica is already servicing an update from a current leader or another vote. I0917 09:19:49.179543 14945 leader_election.cc:310] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82e2633778254c44a45393ac32540f27, abfd3dc05c4a4566bb328f04255eb266; no voters: e501d8ef778246b69b34d86e7395022b I0917 09:19:49.179546 15058 leader_election.cc:310] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e501d8ef778246b69b34d86e7395022b; no voters: 82e2633778254c44a45393ac32540f27, abfd3dc05c4a4566bb328f04255eb266 I0917 09:19:49.179620 15115 raft_consensus.cc:2592] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority I0917 09:19:49.179625 15118 raft_consensus.cc:2644] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 1 FOLLOWER]: Leader election won for term 1 I0917 09:19:49.179775 15118 raft_consensus.cc:667] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 1 LEADER]: Becoming Leader. State: Replica: abfd3dc05c4a4566bb328f04255eb266, State: Running, Role: LEADER I0917 09:19:49.179852 15118 consensus_queue.cc:206] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } I0917 09:19:49.180052 15118 sys_catalog.cc:340] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abfd3dc05c4a4566bb328f04255eb266. Latest consensus state: current_term: 1 leader_uuid: "abfd3dc05c4a4566bb328f04255eb266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:49.180068 15118 sys_catalog.cc:343] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [sys.catalog]: This master's current role is: LEADER I0917 09:19:49.181207 15131 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0917 09:19:49.181344 15131 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0917 09:19:49.212527 14145 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0917 09:19:49.212558 14145 ts_tablet_manager.cc:403] Registered 0 tablets I0917 09:19:49.214807 14145 rpc_server.cc:205] RPC server started. Bound to: 127.13.208.65:39971 I0917 09:19:49.215068 14145 webserver.cc:374] Webserver started at http://127.13.208.65:44321/ using document root and password file I0917 09:19:49.219272 14145 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 27304us I0917 09:19:49.219471 14145 fs_manager.cc:263] Metadata directory not provided I0917 09:19:49.219534 14145 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0917 09:19:49.219615 14145 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0917 09:19:49.219650 14145 server_base.cc:453] Attempting to create new FS layout instead I0917 09:19:49.219997 14145 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/ts-1-root/instance: uuid: "f42c56d2dc6b4faba5a218e79da90951" format_stamp: "Formatted at 2019-09-17 09:19:49 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:49.220685 15192 heartbeater.cc:322] Connected to a master server at 127.13.208.126:33685 I0917 09:19:49.220753 15192 heartbeater.cc:406] Registering TS with master... I0917 09:19:49.220839 15192 heartbeater.cc:455] Master 127.13.208.126:33685 requested a full tablet report, sending... I0917 09:19:49.221128 14963 ts_manager.cc:161] Registered new tserver with Master: c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65:39971) I0917 09:19:49.222581 15191 heartbeater.cc:322] Connected to a master server at 127.13.208.125:37503 I0917 09:19:49.222640 15191 heartbeater.cc:406] Registering TS with master... I0917 09:19:49.222690 15191 heartbeater.cc:455] Master 127.13.208.125:37503 requested a full tablet report, sending... I0917 09:19:49.222697 15190 heartbeater.cc:322] Connected to a master server at 127.13.208.124:39307 I0917 09:19:49.222749 15190 heartbeater.cc:406] Registering TS with master... I0917 09:19:49.222812 15190 heartbeater.cc:455] Master 127.13.208.124:39307 requested a full tablet report, sending... I0917 09:19:49.223019 15021 ts_manager.cc:161] Registered new tserver with Master: c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65:39971) I0917 09:19:49.223078 15078 ts_manager.cc:161] Registered new tserver with Master: c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65:39971) I0917 09:19:49.225159 14145 fs_manager.cc:503] Time spent creating directory manager: real 0.005s user 0.000s sys 0.002s I0917 09:19:49.225493 14145 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0917 09:19:49.227025 14145 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:49.227075 14145 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/ts-1-root uuid: "f42c56d2dc6b4faba5a218e79da90951" format_stamp: "Formatted at 2019-09-17 09:19:49 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:49.227100 14145 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0917 09:19:49.240516 15088 raft_consensus.cc:1184] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 1 FOLLOWER]: Refusing update from remote peer abfd3dc05c4a4566bb328f04255eb266: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0917 09:19:49.240644 15030 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 1 FOLLOWER]: Refusing update from remote peer abfd3dc05c4a4566bb328f04255eb266: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0917 09:19:49.240731 15129 consensus_queue.cc:984] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0917 09:19:49.240901 15129 consensus_queue.cc:984] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0917 09:19:49.241295 15115 sys_catalog.cc:340] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [sys.catalog]: SysCatalogTable state changed. Reason: New leader abfd3dc05c4a4566bb328f04255eb266. Latest consensus state: current_term: 1 leader_uuid: "abfd3dc05c4a4566bb328f04255eb266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:49.241318 15115 sys_catalog.cc:343] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [sys.catalog]: This master's current role is: FOLLOWER I0917 09:19:49.241513 15121 sys_catalog.cc:340] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abfd3dc05c4a4566bb328f04255eb266. Latest consensus state: current_term: 1 leader_uuid: "abfd3dc05c4a4566bb328f04255eb266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:49.241536 15121 sys_catalog.cc:343] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [sys.catalog]: This master's current role is: FOLLOWER I0917 09:19:49.242208 15118 sys_catalog.cc:340] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "abfd3dc05c4a4566bb328f04255eb266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:49.242241 15118 sys_catalog.cc:343] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [sys.catalog]: This master's current role is: LEADER I0917 09:19:49.242264 15115 sys_catalog.cc:340] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "abfd3dc05c4a4566bb328f04255eb266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:49.242278 15115 sys_catalog.cc:343] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [sys.catalog]: This master's current role is: FOLLOWER I0917 09:19:49.242467 15131 catalog_manager.cc:981] Generated new certificate authority record I0917 09:19:49.242517 15118 sys_catalog.cc:340] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "abfd3dc05c4a4566bb328f04255eb266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:49.242534 15118 sys_catalog.cc:343] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [sys.catalog]: This master's current role is: LEADER I0917 09:19:49.242765 15121 sys_catalog.cc:340] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "abfd3dc05c4a4566bb328f04255eb266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:49.242795 15121 sys_catalog.cc:343] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [sys.catalog]: This master's current role is: FOLLOWER I0917 09:19:49.243412 15131 catalog_manager.cc:1106] Loading token signing keys... I0917 09:19:49.245069 15206 mvcc.cc:203] Tried to move safe_time back from 6425444307927605248 to 6425444307681001472. Current Snapshot: MvccSnapshot[committed={T|T < 6425444307927605248}] I0917 09:19:49.248404 14145 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0917 09:19:49.248430 14145 ts_tablet_manager.cc:403] Registered 0 tablets I0917 09:19:49.251417 14145 rpc_server.cc:205] RPC server started. Bound to: 127.13.208.66:38793 I0917 09:19:49.251708 14145 webserver.cc:374] Webserver started at http://127.13.208.66:45521/ using document root and password file I0917 09:19:49.252189 15131 catalog_manager.cc:4435] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266: Generated new TSK 0 I0917 09:19:49.256069 14145 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 27304us I0917 09:19:49.256134 14145 fs_manager.cc:263] Metadata directory not provided I0917 09:19:49.256166 14145 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0917 09:19:49.256220 14145 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0917 09:19:49.256232 14145 server_base.cc:453] Attempting to create new FS layout instead I0917 09:19:49.256501 14145 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/ts-2-root/instance: uuid: "eea10e830a664ea4be74d8cffbe95552" format_stamp: "Formatted at 2019-09-17 09:19:49 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:49.259910 14145 fs_manager.cc:503] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s I0917 09:19:49.261340 14145 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0917 09:19:49.262001 15269 heartbeater.cc:322] Connected to a master server at 127.13.208.124:39307 I0917 09:19:49.262044 15269 heartbeater.cc:406] Registering TS with master... I0917 09:19:49.262106 15271 heartbeater.cc:322] Connected to a master server at 127.13.208.126:33685 I0917 09:19:49.262120 15271 heartbeater.cc:406] Registering TS with master... I0917 09:19:49.262147 15271 heartbeater.cc:455] Master 127.13.208.126:33685 requested a full tablet report, sending... I0917 09:19:49.262220 15269 heartbeater.cc:455] Master 127.13.208.124:39307 requested a full tablet report, sending... I0917 09:19:49.262369 15270 heartbeater.cc:322] Connected to a master server at 127.13.208.125:37503 I0917 09:19:49.262392 15270 heartbeater.cc:406] Registering TS with master... I0917 09:19:49.262447 15270 heartbeater.cc:455] Master 127.13.208.125:37503 requested a full tablet report, sending... I0917 09:19:49.262540 14963 ts_manager.cc:161] Registered new tserver with Master: f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793) I0917 09:19:49.262579 15078 ts_manager.cc:161] Registered new tserver with Master: f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793) I0917 09:19:49.263509 14145 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:49.263550 14145 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/ts-2-root uuid: "eea10e830a664ea4be74d8cffbe95552" format_stamp: "Formatted at 2019-09-17 09:19:49 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:49.263576 14145 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0917 09:19:49.263586 14963 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43002 I0917 09:19:49.263682 15021 ts_manager.cc:161] Registered new tserver with Master: f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793) I0917 09:19:49.297369 14145 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0917 09:19:49.297397 14145 ts_tablet_manager.cc:403] Registered 0 tablets I0917 09:19:49.300029 14145 rpc_server.cc:205] RPC server started. Bound to: 127.13.208.67:42065 I0917 09:19:49.300360 14145 webserver.cc:374] Webserver started at http://127.13.208.67:37125/ using document root and password file I0917 09:19:49.305588 15342 heartbeater.cc:322] Connected to a master server at 127.13.208.125:37503 I0917 09:19:49.305636 15342 heartbeater.cc:406] Registering TS with master... I0917 09:19:49.305718 15342 heartbeater.cc:455] Master 127.13.208.125:37503 requested a full tablet report, sending... I0917 09:19:49.306162 15021 ts_manager.cc:161] Registered new tserver with Master: eea10e830a664ea4be74d8cffbe95552 (127.13.208.67:42065) I0917 09:19:49.306295 15343 heartbeater.cc:322] Connected to a master server at 127.13.208.126:33685 I0917 09:19:49.306324 15343 heartbeater.cc:406] Registering TS with master... I0917 09:19:49.306358 15341 heartbeater.cc:322] Connected to a master server at 127.13.208.124:39307 I0917 09:19:49.306392 15343 heartbeater.cc:455] Master 127.13.208.126:33685 requested a full tablet report, sending... I0917 09:19:49.306401 15341 heartbeater.cc:406] Registering TS with master... I0917 09:19:49.306443 15341 heartbeater.cc:455] Master 127.13.208.124:39307 requested a full tablet report, sending... I0917 09:19:49.306581 14963 ts_manager.cc:161] Registered new tserver with Master: eea10e830a664ea4be74d8cffbe95552 (127.13.208.67:42065) I0917 09:19:49.306718 15078 ts_manager.cc:161] Registered new tserver with Master: eea10e830a664ea4be74d8cffbe95552 (127.13.208.67:42065) I0917 09:19:49.307118 14963 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43008 I0917 09:19:49.307159 14145 internal_mini_cluster.cc:328] 3 TS(s) registered with all masters after 0.006595312s I0917 09:19:49.313504 14963 catalog_manager.cc:1440] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43010: name: "test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } W0917 09:19:49.313601 14963 catalog_manager.cc:1605] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0917 09:19:49.320611 15157 tablet_service.cc:1022] Processing CreateTablet for tablet ac74b319ad54416685f8b9d9506e1d61 (table=test-table [id=03b5e80e6ff342bab99657ab154c5e00]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0917 09:19:49.320691 15156 tablet_service.cc:1022] Processing CreateTablet for tablet f67125c2201a4791ab5756ecc8fba82a (table=test-table [id=03b5e80e6ff342bab99657ab154c5e00]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0917 09:19:49.320827 15156 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet f67125c2201a4791ab5756ecc8fba82a. 1 dirs total, 0 dirs full, 0 dirs failed I0917 09:19:49.320703 15155 tablet_service.cc:1022] Processing CreateTablet for tablet de49f5bae5244941af0020522f04d827 (table=test-table [id=03b5e80e6ff342bab99657ab154c5e00]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0917 09:19:49.320998 15155 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet de49f5bae5244941af0020522f04d827. 1 dirs total, 0 dirs full, 0 dirs failed I0917 09:19:49.322113 15361 tablet_bootstrap.cc:439] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a: Bootstrap starting. I0917 09:19:49.322269 15157 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet ac74b319ad54416685f8b9d9506e1d61. 1 dirs total, 0 dirs full, 0 dirs failed I0917 09:19:49.323768 15361 tablet_bootstrap.cc:590] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a: No blocks or log segments found. Creating new log. I0917 09:19:49.323801 14963 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42996 I0917 09:19:49.326330 15361 tablet_bootstrap.cc:439] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a: No bootstrap required, opened a new log I0917 09:19:49.326475 15361 ts_tablet_manager.cc:1085] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s I0917 09:19:49.326932 15236 tablet_service.cc:1022] Processing CreateTablet for tablet ac74b319ad54416685f8b9d9506e1d61 (table=test-table [id=03b5e80e6ff342bab99657ab154c5e00]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0917 09:19:49.327030 15236 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet ac74b319ad54416685f8b9d9506e1d61. 1 dirs total, 0 dirs full, 0 dirs failed I0917 09:19:49.327220 15234 tablet_service.cc:1022] Processing CreateTablet for tablet de49f5bae5244941af0020522f04d827 (table=test-table [id=03b5e80e6ff342bab99657ab154c5e00]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0917 09:19:49.327343 15235 tablet_service.cc:1022] Processing CreateTablet for tablet f67125c2201a4791ab5756ecc8fba82a (table=test-table [id=03b5e80e6ff342bab99657ab154c5e00]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0917 09:19:49.327404 15234 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet de49f5bae5244941af0020522f04d827. 1 dirs total, 0 dirs full, 0 dirs failed I0917 09:19:49.327548 15235 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet f67125c2201a4791ab5756ecc8fba82a. 1 dirs total, 0 dirs full, 0 dirs failed I0917 09:19:49.327735 15308 tablet_service.cc:1022] Processing CreateTablet for tablet ac74b319ad54416685f8b9d9506e1d61 (table=test-table [id=03b5e80e6ff342bab99657ab154c5e00]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0917 09:19:49.327886 15361 raft_consensus.cc:340] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:49.328135 15361 raft_consensus.cc:366] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0917 09:19:49.328218 15361 raft_consensus.cc:705] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2c8bec4d40141a6b1f470f6142de67a, State: Initialized, Role: FOLLOWER I0917 09:19:49.328234 15306 tablet_service.cc:1022] Processing CreateTablet for tablet de49f5bae5244941af0020522f04d827 (table=test-table [id=03b5e80e6ff342bab99657ab154c5e00]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0917 09:19:49.328578 15308 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet ac74b319ad54416685f8b9d9506e1d61. 1 dirs total, 0 dirs full, 0 dirs failed I0917 09:19:49.328754 15307 tablet_service.cc:1022] Processing CreateTablet for tablet f67125c2201a4791ab5756ecc8fba82a (table=test-table [id=03b5e80e6ff342bab99657ab154c5e00]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0917 09:19:49.328876 15306 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet de49f5bae5244941af0020522f04d827. 1 dirs total, 0 dirs full, 0 dirs failed I0917 09:19:49.329118 15361 consensus_queue.cc:229] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:49.329967 15307 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet f67125c2201a4791ab5756ecc8fba82a. 1 dirs total, 0 dirs full, 0 dirs failed I0917 09:19:49.330567 15192 heartbeater.cc:447] Master 127.13.208.126:33685 was elected leader, sending a full tablet report... I0917 09:19:49.331385 15361 ts_tablet_manager.cc:1113] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s I0917 09:19:49.332286 15361 tablet_bootstrap.cc:439] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a: Bootstrap starting. I0917 09:19:49.332923 15364 tablet_bootstrap.cc:439] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951: Bootstrap starting. I0917 09:19:49.334882 15364 tablet_bootstrap.cc:590] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951: No blocks or log segments found. Creating new log. I0917 09:19:49.335032 15361 tablet_bootstrap.cc:590] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a: No blocks or log segments found. Creating new log. I0917 09:19:49.337013 15361 tablet_bootstrap.cc:439] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a: No bootstrap required, opened a new log I0917 09:19:49.337139 15361 ts_tablet_manager.cc:1085] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s I0917 09:19:49.338449 15364 tablet_bootstrap.cc:439] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951: No bootstrap required, opened a new log I0917 09:19:49.338615 15364 ts_tablet_manager.cc:1085] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s I0917 09:19:49.339516 15364 raft_consensus.cc:340] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:49.339558 15364 raft_consensus.cc:366] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0917 09:19:49.339583 15364 raft_consensus.cc:705] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f42c56d2dc6b4faba5a218e79da90951, State: Initialized, Role: FOLLOWER I0917 09:19:49.339721 15361 raft_consensus.cc:340] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:49.339866 15364 consensus_queue.cc:229] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:49.340355 15271 heartbeater.cc:447] Master 127.13.208.126:33685 was elected leader, sending a full tablet report... I0917 09:19:49.340445 15361 raft_consensus.cc:366] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0917 09:19:49.340503 15361 raft_consensus.cc:705] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2c8bec4d40141a6b1f470f6142de67a, State: Initialized, Role: FOLLOWER I0917 09:19:49.340452 15368 tablet_bootstrap.cc:439] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552: Bootstrap starting. I0917 09:19:49.340983 15361 consensus_queue.cc:229] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:49.341074 15364 ts_tablet_manager.cc:1113] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s I0917 09:19:49.341367 15364 tablet_bootstrap.cc:439] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951: Bootstrap starting. I0917 09:19:49.341289 15361 ts_tablet_manager.cc:1113] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s I0917 09:19:49.341979 15361 tablet_bootstrap.cc:439] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a: Bootstrap starting. I0917 09:19:49.342772 15364 tablet_bootstrap.cc:590] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951: No blocks or log segments found. Creating new log. I0917 09:19:49.342857 15368 tablet_bootstrap.cc:590] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552: No blocks or log segments found. Creating new log. I0917 09:19:49.343015 15361 tablet_bootstrap.cc:590] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a: No blocks or log segments found. Creating new log. I0917 09:19:49.343724 15361 tablet_bootstrap.cc:439] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a: No bootstrap required, opened a new log I0917 09:19:49.343761 15364 tablet_bootstrap.cc:439] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951: No bootstrap required, opened a new log I0917 09:19:49.343791 15361 ts_tablet_manager.cc:1085] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s I0917 09:19:49.343860 15364 ts_tablet_manager.cc:1085] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s I0917 09:19:49.343909 15368 tablet_bootstrap.cc:439] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552: No bootstrap required, opened a new log I0917 09:19:49.344023 15368 ts_tablet_manager.cc:1085] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s I0917 09:19:49.344257 15368 raft_consensus.cc:340] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:49.344276 15368 raft_consensus.cc:366] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0917 09:19:49.344295 15368 raft_consensus.cc:705] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea10e830a664ea4be74d8cffbe95552, State: Initialized, Role: FOLLOWER I0917 09:19:49.344327 15361 raft_consensus.cc:340] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } I0917 09:19:49.344411 15361 raft_consensus.cc:366] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0917 09:19:49.344472 15361 raft_consensus.cc:705] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2c8bec4d40141a6b1f470f6142de67a, State: Initialized, Role: FOLLOWER I0917 09:19:49.344496 15364 raft_consensus.cc:340] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:49.344512 15364 raft_consensus.cc:366] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0917 09:19:49.344523 15364 raft_consensus.cc:705] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f42c56d2dc6b4faba5a218e79da90951, State: Initialized, Role: FOLLOWER I0917 09:19:49.344607 15361 consensus_queue.cc:229] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } I0917 09:19:49.344626 15368 consensus_queue.cc:229] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:49.344671 15364 consensus_queue.cc:229] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:49.345125 15343 heartbeater.cc:447] Master 127.13.208.126:33685 was elected leader, sending a full tablet report... I0917 09:19:49.345340 15368 ts_tablet_manager.cc:1113] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0917 09:19:49.345468 15368 tablet_bootstrap.cc:439] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552: Bootstrap starting. I0917 09:19:49.345626 15361 ts_tablet_manager.cc:1113] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s I0917 09:19:49.345751 15364 ts_tablet_manager.cc:1113] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s I0917 09:19:49.345888 15364 tablet_bootstrap.cc:439] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: Bootstrap starting. I0917 09:19:49.347904 15368 tablet_bootstrap.cc:590] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552: No blocks or log segments found. Creating new log. I0917 09:19:49.347973 15364 tablet_bootstrap.cc:590] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: No blocks or log segments found. Creating new log. I0917 09:19:49.348762 15364 tablet_bootstrap.cc:439] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: No bootstrap required, opened a new log I0917 09:19:49.348773 15368 tablet_bootstrap.cc:439] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552: No bootstrap required, opened a new log I0917 09:19:49.348817 15368 ts_tablet_manager.cc:1085] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s I0917 09:19:49.348821 15364 ts_tablet_manager.cc:1085] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s I0917 09:19:49.349141 15364 raft_consensus.cc:340] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } I0917 09:19:49.349151 15368 raft_consensus.cc:340] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:49.349161 15364 raft_consensus.cc:366] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0917 09:19:49.349165 15368 raft_consensus.cc:366] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0917 09:19:49.349174 15364 raft_consensus.cc:705] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f42c56d2dc6b4faba5a218e79da90951, State: Initialized, Role: FOLLOWER I0917 09:19:49.349175 15368 raft_consensus.cc:705] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea10e830a664ea4be74d8cffbe95552, State: Initialized, Role: FOLLOWER I0917 09:19:49.349300 15364 consensus_queue.cc:229] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } I0917 09:19:49.349351 15368 consensus_queue.cc:229] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:49.349758 15368 ts_tablet_manager.cc:1113] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s I0917 09:19:49.349889 15364 ts_tablet_manager.cc:1113] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s I0917 09:19:49.349900 15368 tablet_bootstrap.cc:439] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552: Bootstrap starting. I0917 09:19:49.350693 15372 raft_consensus.cc:472] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0917 09:19:49.351013 15372 raft_consensus.cc:494] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:49.351456 15372 leader_election.cc:296] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793), c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65:39971) I0917 09:19:49.351675 15368 tablet_bootstrap.cc:590] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552: No blocks or log segments found. Creating new log. I0917 09:19:49.352344 15368 tablet_bootstrap.cc:439] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552: No bootstrap required, opened a new log I0917 09:19:49.352380 15368 ts_tablet_manager.cc:1085] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s I0917 09:19:49.352630 15368 raft_consensus.cc:340] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } I0917 09:19:49.352645 15368 raft_consensus.cc:366] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0917 09:19:49.352653 15368 raft_consensus.cc:705] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea10e830a664ea4be74d8cffbe95552, State: Initialized, Role: FOLLOWER I0917 09:19:49.353798 15368 consensus_queue.cc:229] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } I0917 09:19:49.354017 15368 ts_tablet_manager.cc:1113] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s I0917 09:19:49.357084 15243 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f67125c2201a4791ab5756ecc8fba82a" candidate_uuid: "eea10e830a664ea4be74d8cffbe95552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f42c56d2dc6b4faba5a218e79da90951" is_pre_election: true I0917 09:19:49.357117 15243 raft_consensus.cc:2350] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eea10e830a664ea4be74d8cffbe95552 in term 0. I0917 09:19:49.357213 15279 leader_election.cc:310] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eea10e830a664ea4be74d8cffbe95552, f42c56d2dc6b4faba5a218e79da90951; no voters: I0917 09:19:49.357264 15372 raft_consensus.cc:2644] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0917 09:19:49.357349 15372 raft_consensus.cc:472] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0917 09:19:49.357389 15372 raft_consensus.cc:2890] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 0 FOLLOWER]: Advancing to term 1 I0917 09:19:49.357496 15167 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f67125c2201a4791ab5756ecc8fba82a" candidate_uuid: "eea10e830a664ea4be74d8cffbe95552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2c8bec4d40141a6b1f470f6142de67a" is_pre_election: true I0917 09:19:49.357529 15167 raft_consensus.cc:2350] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eea10e830a664ea4be74d8cffbe95552 in term 0. I0917 09:19:49.358325 15372 raft_consensus.cc:494] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:49.358465 15372 leader_election.cc:296] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [CANDIDATE]: Term 1 election: Requested vote from peers f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793), c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65:39971) I0917 09:19:49.358587 15167 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f67125c2201a4791ab5756ecc8fba82a" candidate_uuid: "eea10e830a664ea4be74d8cffbe95552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2c8bec4d40141a6b1f470f6142de67a" I0917 09:19:49.358618 15167 raft_consensus.cc:2890] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 0 FOLLOWER]: Advancing to term 1 I0917 09:19:49.358651 15243 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f67125c2201a4791ab5756ecc8fba82a" candidate_uuid: "eea10e830a664ea4be74d8cffbe95552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f42c56d2dc6b4faba5a218e79da90951" I0917 09:19:49.358678 15243 raft_consensus.cc:2890] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 0 FOLLOWER]: Advancing to term 1 I0917 09:19:49.359809 15167 raft_consensus.cc:2350] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eea10e830a664ea4be74d8cffbe95552 in term 1. I0917 09:19:49.359853 15243 raft_consensus.cc:2350] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eea10e830a664ea4be74d8cffbe95552 in term 1. I0917 09:19:49.360007 15280 leader_election.cc:310] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: c2c8bec4d40141a6b1f470f6142de67a, eea10e830a664ea4be74d8cffbe95552, f42c56d2dc6b4faba5a218e79da90951; no voters: I0917 09:19:49.360069 15372 raft_consensus.cc:2644] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 1 FOLLOWER]: Leader election won for term 1 I0917 09:19:49.360174 15372 raft_consensus.cc:667] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 1 LEADER]: Becoming Leader. State: Replica: eea10e830a664ea4be74d8cffbe95552, State: Running, Role: LEADER I0917 09:19:49.360267 15372 consensus_queue.cc:206] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:49.361042 14964 catalog_manager.cc:4120] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 reported cstate change: term changed from 0 to 1, leader changed from to eea10e830a664ea4be74d8cffbe95552 (127.13.208.67). New cstate: current_term: 1 leader_uuid: "eea10e830a664ea4be74d8cffbe95552" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } health_report { overall_health: HEALTHY } } } I0917 09:19:49.365869 15363 raft_consensus.cc:472] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0917 09:19:49.365944 15363 raft_consensus.cc:494] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } I0917 09:19:49.366041 15363 leader_election.cc:296] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eea10e830a664ea4be74d8cffbe95552 (127.13.208.67:42065), f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793) I0917 09:19:49.369814 15243 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ac74b319ad54416685f8b9d9506e1d61" candidate_uuid: "c2c8bec4d40141a6b1f470f6142de67a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f42c56d2dc6b4faba5a218e79da90951" is_pre_election: true I0917 09:19:49.369846 15318 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ac74b319ad54416685f8b9d9506e1d61" candidate_uuid: "c2c8bec4d40141a6b1f470f6142de67a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eea10e830a664ea4be74d8cffbe95552" is_pre_election: true I0917 09:19:49.369866 15243 raft_consensus.cc:2350] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2c8bec4d40141a6b1f470f6142de67a in term 0. I0917 09:19:49.369884 15318 raft_consensus.cc:2350] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2c8bec4d40141a6b1f470f6142de67a in term 0. I0917 09:19:49.369984 15125 leader_election.cc:310] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2c8bec4d40141a6b1f470f6142de67a, f42c56d2dc6b4faba5a218e79da90951; no voters: I0917 09:19:49.370055 15363 raft_consensus.cc:2644] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 0 FOLLOWER]: Leader pre-election won for term 1 I0917 09:19:49.370076 15363 raft_consensus.cc:472] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0917 09:19:49.370084 15363 raft_consensus.cc:2890] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 0 FOLLOWER]: Advancing to term 1 I0917 09:19:49.370744 15363 raft_consensus.cc:494] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } I0917 09:19:49.370818 15363 leader_election.cc:296] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [CANDIDATE]: Term 1 election: Requested vote from peers eea10e830a664ea4be74d8cffbe95552 (127.13.208.67:42065), f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793) I0917 09:19:49.370996 15243 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ac74b319ad54416685f8b9d9506e1d61" candidate_uuid: "c2c8bec4d40141a6b1f470f6142de67a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f42c56d2dc6b4faba5a218e79da90951" I0917 09:19:49.371006 15318 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ac74b319ad54416685f8b9d9506e1d61" candidate_uuid: "c2c8bec4d40141a6b1f470f6142de67a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eea10e830a664ea4be74d8cffbe95552" I0917 09:19:49.371033 15243 raft_consensus.cc:2890] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 0 FOLLOWER]: Advancing to term 1 I0917 09:19:49.371035 15318 raft_consensus.cc:2890] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [term 0 FOLLOWER]: Advancing to term 1 I0917 09:19:49.372058 15318 raft_consensus.cc:2350] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2c8bec4d40141a6b1f470f6142de67a in term 1. I0917 09:19:49.372083 15243 raft_consensus.cc:2350] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2c8bec4d40141a6b1f470f6142de67a in term 1. I0917 09:19:49.372217 15125 leader_election.cc:310] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2c8bec4d40141a6b1f470f6142de67a, f42c56d2dc6b4faba5a218e79da90951; no voters: I0917 09:19:49.372298 15363 raft_consensus.cc:2644] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 1 FOLLOWER]: Leader election won for term 1 I0917 09:19:49.372398 15363 raft_consensus.cc:667] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 1 LEADER]: Becoming Leader. State: Replica: c2c8bec4d40141a6b1f470f6142de67a, State: Running, Role: LEADER I0917 09:19:49.372457 15363 consensus_queue.cc:206] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } I0917 09:19:49.373313 14964 catalog_manager.cc:4120] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a reported cstate change: term changed from 0 to 1, leader changed from to c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65). New cstate: current_term: 1 leader_uuid: "c2c8bec4d40141a6b1f470f6142de67a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } health_report { overall_health: UNKNOWN } } } I0917 09:19:49.381217 15376 raft_consensus.cc:472] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0917 09:19:49.381314 15376 raft_consensus.cc:494] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:49.381395 15376 leader_election.cc:296] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65:39971), f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793) I0917 09:19:49.381701 15167 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "de49f5bae5244941af0020522f04d827" candidate_uuid: "eea10e830a664ea4be74d8cffbe95552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2c8bec4d40141a6b1f470f6142de67a" is_pre_election: true I0917 09:19:49.381743 15167 raft_consensus.cc:2350] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eea10e830a664ea4be74d8cffbe95552 in term 0. I0917 09:19:49.381853 15280 leader_election.cc:310] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2c8bec4d40141a6b1f470f6142de67a, eea10e830a664ea4be74d8cffbe95552; no voters: I0917 09:19:49.381911 15376 raft_consensus.cc:2644] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0917 09:19:49.381927 15376 raft_consensus.cc:472] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0917 09:19:49.381934 15376 raft_consensus.cc:2890] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [term 0 FOLLOWER]: Advancing to term 1 I0917 09:19:49.382342 15243 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "de49f5bae5244941af0020522f04d827" candidate_uuid: "eea10e830a664ea4be74d8cffbe95552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f42c56d2dc6b4faba5a218e79da90951" is_pre_election: true I0917 09:19:49.382376 15243 raft_consensus.cc:2350] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eea10e830a664ea4be74d8cffbe95552 in term 0. I0917 09:19:49.382699 15376 raft_consensus.cc:494] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:49.382769 15376 leader_election.cc:296] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [CANDIDATE]: Term 1 election: Requested vote from peers c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65:39971), f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793) I0917 09:19:49.382899 15243 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "de49f5bae5244941af0020522f04d827" candidate_uuid: "eea10e830a664ea4be74d8cffbe95552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f42c56d2dc6b4faba5a218e79da90951" I0917 09:19:49.382921 15243 raft_consensus.cc:2890] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [term 0 FOLLOWER]: Advancing to term 1 I0917 09:19:49.383030 15167 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "de49f5bae5244941af0020522f04d827" candidate_uuid: "eea10e830a664ea4be74d8cffbe95552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2c8bec4d40141a6b1f470f6142de67a" I0917 09:19:49.383057 15167 raft_consensus.cc:2890] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [term 0 FOLLOWER]: Advancing to term 1 I0917 09:19:49.384040 15243 raft_consensus.cc:2350] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eea10e830a664ea4be74d8cffbe95552 in term 1. I0917 09:19:49.384220 15279 leader_election.cc:310] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eea10e830a664ea4be74d8cffbe95552, f42c56d2dc6b4faba5a218e79da90951; no voters: I0917 09:19:49.384291 15376 raft_consensus.cc:2644] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [term 1 FOLLOWER]: Leader election won for term 1 I0917 09:19:49.384356 15376 raft_consensus.cc:667] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [term 1 LEADER]: Becoming Leader. State: Replica: eea10e830a664ea4be74d8cffbe95552, State: Running, Role: LEADER I0917 09:19:49.384310 15167 raft_consensus.cc:2350] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eea10e830a664ea4be74d8cffbe95552 in term 1. I0917 09:19:49.384558 15376 consensus_queue.cc:206] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:49.385442 14964 catalog_manager.cc:4120] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 reported cstate change: term changed from 0 to 1, leader changed from to eea10e830a664ea4be74d8cffbe95552 (127.13.208.67). New cstate: current_term: 1 leader_uuid: "eea10e830a664ea4be74d8cffbe95552" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } health_report { overall_health: HEALTHY } } } I0917 09:19:49.410693 15167 raft_consensus.cc:1184] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 1 FOLLOWER]: Refusing update from remote peer eea10e830a664ea4be74d8cffbe95552: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0917 09:19:49.410797 15243 raft_consensus.cc:1184] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 1 FOLLOWER]: Refusing update from remote peer eea10e830a664ea4be74d8cffbe95552: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0917 09:19:49.410892 15242 raft_consensus.cc:1184] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [term 1 FOLLOWER]: Refusing update from remote peer eea10e830a664ea4be74d8cffbe95552: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0917 09:19:49.411023 15376 consensus_queue.cc:984] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0917 09:19:49.411176 15372 consensus_queue.cc:984] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0917 09:19:49.411206 15167 raft_consensus.cc:1184] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [term 1 FOLLOWER]: Refusing update from remote peer eea10e830a664ea4be74d8cffbe95552: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0917 09:19:49.411581 15376 consensus_queue.cc:984] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0917 09:19:49.411864 15385 consensus_queue.cc:984] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0917 09:19:49.412118 15241 raft_consensus.cc:1184] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 1 FOLLOWER]: Refusing update from remote peer c2c8bec4d40141a6b1f470f6142de67a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0917 09:19:49.412605 15378 consensus_queue.cc:984] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0917 09:19:49.413458 15318 raft_consensus.cc:1184] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [term 1 FOLLOWER]: Refusing update from remote peer c2c8bec4d40141a6b1f470f6142de67a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0917 09:19:49.414145 15378 consensus_queue.cc:984] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0917 09:19:49.430096 15167 tablet_service.cc:1440] Received LeaderStepDown RPC: tablet_id: "ac74b319ad54416685f8b9d9506e1d61" dest_uuid: "c2c8bec4d40141a6b1f470f6142de67a" from {username='slave'} at 127.0.0.1:45876 I0917 09:19:49.430142 15167 raft_consensus.cc:569] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 1 LEADER]: Received request to step down I0917 09:19:49.430151 15167 raft_consensus.cc:2885] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 1 LEADER]: Stepping down as leader of term 1 I0917 09:19:49.430158 15167 raft_consensus.cc:705] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c2c8bec4d40141a6b1f470f6142de67a, State: Running, Role: LEADER I0917 09:19:49.430244 15167 consensus_queue.cc:229] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } I0917 09:19:49.430286 15167 raft_consensus.cc:2890] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 1 FOLLOWER]: Advancing to term 2 I0917 09:19:49.618610 15404 raft_consensus.cc:472] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c2c8bec4d40141a6b1f470f6142de67a) I0917 09:19:49.618710 15404 raft_consensus.cc:494] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } I0917 09:19:49.618824 15404 leader_election.cc:296] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eea10e830a664ea4be74d8cffbe95552 (127.13.208.67:42065), c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65:39971) I0917 09:19:49.624220 15385 raft_consensus.cc:472] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c2c8bec4d40141a6b1f470f6142de67a) I0917 09:19:49.624485 15318 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ac74b319ad54416685f8b9d9506e1d61" candidate_uuid: "f42c56d2dc6b4faba5a218e79da90951" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "eea10e830a664ea4be74d8cffbe95552" is_pre_election: true I0917 09:19:49.624558 15385 raft_consensus.cc:494] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } I0917 09:19:49.624614 15318 raft_consensus.cc:2350] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f42c56d2dc6b4faba5a218e79da90951 in term 1. I0917 09:19:49.624729 15167 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ac74b319ad54416685f8b9d9506e1d61" candidate_uuid: "f42c56d2dc6b4faba5a218e79da90951" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c2c8bec4d40141a6b1f470f6142de67a" is_pre_election: true I0917 09:19:49.624783 15167 raft_consensus.cc:2350] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f42c56d2dc6b4faba5a218e79da90951 in term 2. I0917 09:19:49.624825 15202 leader_election.cc:310] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eea10e830a664ea4be74d8cffbe95552, f42c56d2dc6b4faba5a218e79da90951; no voters: I0917 09:19:49.624999 15385 leader_election.cc:296] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65:39971), f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793) I0917 09:19:49.625080 15404 raft_consensus.cc:2644] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0917 09:19:49.625149 15404 raft_consensus.cc:472] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c2c8bec4d40141a6b1f470f6142de67a) I0917 09:19:49.625160 15243 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ac74b319ad54416685f8b9d9506e1d61" candidate_uuid: "eea10e830a664ea4be74d8cffbe95552" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f42c56d2dc6b4faba5a218e79da90951" is_pre_election: true I0917 09:19:49.625165 15404 raft_consensus.cc:2890] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 1 FOLLOWER]: Advancing to term 2 I0917 09:19:49.625756 15167 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ac74b319ad54416685f8b9d9506e1d61" candidate_uuid: "eea10e830a664ea4be74d8cffbe95552" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c2c8bec4d40141a6b1f470f6142de67a" is_pre_election: true I0917 09:19:49.625828 15167 raft_consensus.cc:2350] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eea10e830a664ea4be74d8cffbe95552 in term 2. I0917 09:19:49.626008 15280 leader_election.cc:310] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2c8bec4d40141a6b1f470f6142de67a, eea10e830a664ea4be74d8cffbe95552; no voters: I0917 09:19:49.626109 15385 raft_consensus.cc:2644] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0917 09:19:49.626132 15385 raft_consensus.cc:472] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c2c8bec4d40141a6b1f470f6142de67a) I0917 09:19:49.626140 15385 raft_consensus.cc:2890] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [term 1 FOLLOWER]: Advancing to term 2 I0917 09:19:49.626901 15404 raft_consensus.cc:494] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } I0917 09:19:49.626968 15243 raft_consensus.cc:2280] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eea10e830a664ea4be74d8cffbe95552 in current term 2: Already voted for candidate f42c56d2dc6b4faba5a218e79da90951 in this term. I0917 09:19:49.627009 15404 leader_election.cc:296] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [CANDIDATE]: Term 2 election: Requested vote from peers eea10e830a664ea4be74d8cffbe95552 (127.13.208.67:42065), c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65:39971) I0917 09:19:49.627128 15167 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ac74b319ad54416685f8b9d9506e1d61" candidate_uuid: "f42c56d2dc6b4faba5a218e79da90951" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c2c8bec4d40141a6b1f470f6142de67a" I0917 09:19:49.627367 15385 raft_consensus.cc:494] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } I0917 09:19:49.627466 15318 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ac74b319ad54416685f8b9d9506e1d61" candidate_uuid: "f42c56d2dc6b4faba5a218e79da90951" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "eea10e830a664ea4be74d8cffbe95552" I0917 09:19:49.627497 15318 raft_consensus.cc:2280] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f42c56d2dc6b4faba5a218e79da90951 in current term 2: Already voted for candidate eea10e830a664ea4be74d8cffbe95552 in this term. I0917 09:19:49.627715 15166 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ac74b319ad54416685f8b9d9506e1d61" candidate_uuid: "eea10e830a664ea4be74d8cffbe95552" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c2c8bec4d40141a6b1f470f6142de67a" I0917 09:19:49.627748 15243 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ac74b319ad54416685f8b9d9506e1d61" candidate_uuid: "eea10e830a664ea4be74d8cffbe95552" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f42c56d2dc6b4faba5a218e79da90951" I0917 09:19:49.627768 15243 raft_consensus.cc:2280] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate eea10e830a664ea4be74d8cffbe95552 in current term 2: Already voted for candidate f42c56d2dc6b4faba5a218e79da90951 in this term. I0917 09:19:49.627712 15385 leader_election.cc:296] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [CANDIDATE]: Term 2 election: Requested vote from peers c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65:39971), f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793) I0917 09:19:49.628332 15167 raft_consensus.cc:2350] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f42c56d2dc6b4faba5a218e79da90951 in term 2. I0917 09:19:49.628389 15166 raft_consensus.cc:2325] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate eea10e830a664ea4be74d8cffbe95552 for term 2 because replica is already servicing an update from a current leader or another vote. I0917 09:19:49.628512 15280 leader_election.cc:310] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eea10e830a664ea4be74d8cffbe95552; no voters: c2c8bec4d40141a6b1f470f6142de67a, f42c56d2dc6b4faba5a218e79da90951 I0917 09:19:49.628526 15200 leader_election.cc:310] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c2c8bec4d40141a6b1f470f6142de67a, f42c56d2dc6b4faba5a218e79da90951; no voters: eea10e830a664ea4be74d8cffbe95552 I0917 09:19:49.628661 15404 raft_consensus.cc:2644] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 2 FOLLOWER]: Leader election won for term 2 I0917 09:19:49.628684 15385 raft_consensus.cc:2592] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority I0917 09:19:49.628814 15404 raft_consensus.cc:667] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 2 LEADER]: Becoming Leader. State: Replica: f42c56d2dc6b4faba5a218e79da90951, State: Running, Role: LEADER I0917 09:19:49.628934 15404 consensus_queue.cc:206] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } I0917 09:19:49.629626 14957 catalog_manager.cc:4120] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 reported cstate change: term changed from 1 to 2, leader changed from c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65) to f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66). New cstate: current_term: 2 leader_uuid: "f42c56d2dc6b4faba5a218e79da90951" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } health_report { overall_health: HEALTHY } } } I0917 09:19:49.634565 15243 tablet_service.cc:1440] Received LeaderStepDown RPC: tablet_id: "ac74b319ad54416685f8b9d9506e1d61" dest_uuid: "f42c56d2dc6b4faba5a218e79da90951" from {username='slave'} at 127.0.0.1:45156 I0917 09:19:49.634600 15243 raft_consensus.cc:569] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 2 LEADER]: Received request to step down I0917 09:19:49.634605 15243 raft_consensus.cc:2885] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 2 LEADER]: Stepping down as leader of term 2 I0917 09:19:49.634610 15243 raft_consensus.cc:705] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f42c56d2dc6b4faba5a218e79da90951, State: Running, Role: LEADER I0917 09:19:49.634678 15243 consensus_queue.cc:229] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } I0917 09:19:49.634703 15243 raft_consensus.cc:2890] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 2 FOLLOWER]: Advancing to term 3 I0917 09:19:49.839071 15318 tablet_service.cc:1440] Received LeaderStepDown RPC: tablet_id: "f67125c2201a4791ab5756ecc8fba82a" dest_uuid: "eea10e830a664ea4be74d8cffbe95552" from {username='slave'} at 127.0.0.1:58758 I0917 09:19:49.839108 15318 raft_consensus.cc:569] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 1 LEADER]: Received request to step down I0917 09:19:49.839113 15318 raft_consensus.cc:2885] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 1 LEADER]: Stepping down as leader of term 1 I0917 09:19:49.839118 15318 raft_consensus.cc:705] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eea10e830a664ea4be74d8cffbe95552, State: Running, Role: LEADER I0917 09:19:49.839195 15318 consensus_queue.cc:229] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:49.839236 15318 raft_consensus.cc:2890] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 1 FOLLOWER]: Advancing to term 2 I0917 09:19:49.877743 15385 raft_consensus.cc:472] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0917 09:19:49.877871 15385 raft_consensus.cc:494] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } I0917 09:19:49.877971 15385 leader_election.cc:296] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65:39971), f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793) I0917 09:19:49.878230 15243 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ac74b319ad54416685f8b9d9506e1d61" candidate_uuid: "eea10e830a664ea4be74d8cffbe95552" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f42c56d2dc6b4faba5a218e79da90951" is_pre_election: true I0917 09:19:49.878238 15166 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ac74b319ad54416685f8b9d9506e1d61" candidate_uuid: "eea10e830a664ea4be74d8cffbe95552" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c2c8bec4d40141a6b1f470f6142de67a" is_pre_election: true I0917 09:19:49.878285 15166 raft_consensus.cc:2350] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eea10e830a664ea4be74d8cffbe95552 in term 2. I0917 09:19:49.878293 15243 raft_consensus.cc:2297] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eea10e830a664ea4be74d8cffbe95552 for term 3 because replica has last-logged OpId of term: 2 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2. I0917 09:19:49.878415 15280 leader_election.cc:310] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2c8bec4d40141a6b1f470f6142de67a, eea10e830a664ea4be74d8cffbe95552; no voters: I0917 09:19:49.878515 15385 raft_consensus.cc:2644] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [term 2 FOLLOWER]: Leader pre-election won for term 3 I0917 09:19:49.878535 15385 raft_consensus.cc:472] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0917 09:19:49.878540 15385 raft_consensus.cc:2890] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [term 2 FOLLOWER]: Advancing to term 3 I0917 09:19:49.879340 15385 raft_consensus.cc:494] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } I0917 09:19:49.879418 15385 leader_election.cc:296] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [CANDIDATE]: Term 3 election: Requested vote from peers c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65:39971), f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793) I0917 09:19:49.879654 15243 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ac74b319ad54416685f8b9d9506e1d61" candidate_uuid: "eea10e830a664ea4be74d8cffbe95552" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f42c56d2dc6b4faba5a218e79da90951" I0917 09:19:49.879680 15166 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ac74b319ad54416685f8b9d9506e1d61" candidate_uuid: "eea10e830a664ea4be74d8cffbe95552" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c2c8bec4d40141a6b1f470f6142de67a" I0917 09:19:49.879801 15243 raft_consensus.cc:2297] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eea10e830a664ea4be74d8cffbe95552 for term 3 because replica has last-logged OpId of term: 2 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2. I0917 09:19:49.879812 15166 raft_consensus.cc:2890] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 2 FOLLOWER]: Advancing to term 3 I0917 09:19:49.880451 15166 raft_consensus.cc:2350] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eea10e830a664ea4be74d8cffbe95552 in term 3. I0917 09:19:49.880671 15280 leader_election.cc:310] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c2c8bec4d40141a6b1f470f6142de67a, eea10e830a664ea4be74d8cffbe95552; no voters: f42c56d2dc6b4faba5a218e79da90951 I0917 09:19:49.880739 15385 raft_consensus.cc:2644] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [term 3 FOLLOWER]: Leader election won for term 3 I0917 09:19:49.880769 15385 raft_consensus.cc:667] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [term 3 LEADER]: Becoming Leader. State: Replica: eea10e830a664ea4be74d8cffbe95552, State: Running, Role: LEADER I0917 09:19:49.880870 15385 consensus_queue.cc:206] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } I0917 09:19:49.881628 14958 catalog_manager.cc:4120] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 reported cstate change: term changed from 2 to 3, leader changed from f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66) to eea10e830a664ea4be74d8cffbe95552 (127.13.208.67). New cstate: current_term: 3 leader_uuid: "eea10e830a664ea4be74d8cffbe95552" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } health_report { overall_health: UNKNOWN } } } I0917 09:19:49.966135 15166 raft_consensus.cc:1184] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 3 FOLLOWER]: Refusing update from remote peer eea10e830a664ea4be74d8cffbe95552: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch) I0917 09:19:49.966394 15385 consensus_queue.cc:984] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0917 09:19:49.967620 15243 raft_consensus.cc:1184] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 3 FOLLOWER]: Refusing update from remote peer eea10e830a664ea4be74d8cffbe95552: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 3. (term mismatch) I0917 09:19:49.967667 15243 pending_rounds.cc:78] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: Aborting all transactions after (but not including) 2 I0917 09:19:49.967676 15243 pending_rounds.cc:99] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: Aborting uncommitted NO_OP operation due to leader change: 2.3 I0917 09:19:49.967692 15243 raft_consensus.cc:2727] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader I0917 09:19:49.968057 15385 consensus_queue.cc:984] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0917 09:19:50.022902 15408 raft_consensus.cc:472] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eea10e830a664ea4be74d8cffbe95552) I0917 09:19:50.023038 15408 raft_consensus.cc:494] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:50.023151 15408 leader_election.cc:296] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793), eea10e830a664ea4be74d8cffbe95552 (127.13.208.67:42065) I0917 09:19:50.023455 15243 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f67125c2201a4791ab5756ecc8fba82a" candidate_uuid: "c2c8bec4d40141a6b1f470f6142de67a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f42c56d2dc6b4faba5a218e79da90951" is_pre_election: true I0917 09:19:50.023470 15318 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f67125c2201a4791ab5756ecc8fba82a" candidate_uuid: "c2c8bec4d40141a6b1f470f6142de67a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "eea10e830a664ea4be74d8cffbe95552" is_pre_election: true I0917 09:19:50.023490 15243 raft_consensus.cc:2311] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c2c8bec4d40141a6b1f470f6142de67a for term 2 because replica is either leader or believes a valid leader to be alive. I0917 09:19:50.023504 15318 raft_consensus.cc:2350] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2c8bec4d40141a6b1f470f6142de67a in term 2. I0917 09:19:50.023627 15128 leader_election.cc:310] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2c8bec4d40141a6b1f470f6142de67a, eea10e830a664ea4be74d8cffbe95552; no voters: I0917 09:19:50.023695 15408 raft_consensus.cc:2644] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 1 FOLLOWER]: Leader pre-election won for term 2 I0917 09:19:50.023730 15408 raft_consensus.cc:472] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 1 FOLLOWER]: Starting leader election (detected failure of leader eea10e830a664ea4be74d8cffbe95552) I0917 09:19:50.023744 15408 raft_consensus.cc:2890] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 1 FOLLOWER]: Advancing to term 2 I0917 09:19:50.024494 15408 raft_consensus.cc:494] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:50.024567 15408 leader_election.cc:296] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [CANDIDATE]: Term 2 election: Requested vote from peers f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793), eea10e830a664ea4be74d8cffbe95552 (127.13.208.67:42065) I0917 09:19:50.024786 15318 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f67125c2201a4791ab5756ecc8fba82a" candidate_uuid: "c2c8bec4d40141a6b1f470f6142de67a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "eea10e830a664ea4be74d8cffbe95552" I0917 09:19:50.024789 15243 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f67125c2201a4791ab5756ecc8fba82a" candidate_uuid: "c2c8bec4d40141a6b1f470f6142de67a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f42c56d2dc6b4faba5a218e79da90951" I0917 09:19:50.024825 15243 raft_consensus.cc:2311] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c2c8bec4d40141a6b1f470f6142de67a for term 2 because replica is either leader or believes a valid leader to be alive. I0917 09:19:50.025463 15318 raft_consensus.cc:2350] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2c8bec4d40141a6b1f470f6142de67a in term 2. I0917 09:19:50.025648 15128 leader_election.cc:310] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c2c8bec4d40141a6b1f470f6142de67a, eea10e830a664ea4be74d8cffbe95552; no voters: f42c56d2dc6b4faba5a218e79da90951 I0917 09:19:50.025709 15408 raft_consensus.cc:2644] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 2 FOLLOWER]: Leader election won for term 2 I0917 09:19:50.025820 15408 raft_consensus.cc:667] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 2 LEADER]: Becoming Leader. State: Replica: c2c8bec4d40141a6b1f470f6142de67a, State: Running, Role: LEADER I0917 09:19:50.025902 15408 consensus_queue.cc:206] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:50.026648 14957 catalog_manager.cc:4120] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a reported cstate change: term changed from 1 to 2, leader changed from eea10e830a664ea4be74d8cffbe95552 (127.13.208.67) to c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65). New cstate: current_term: 2 leader_uuid: "c2c8bec4d40141a6b1f470f6142de67a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } health_report { overall_health: UNKNOWN } } } I0917 09:19:50.038913 15404 raft_consensus.cc:472] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eea10e830a664ea4be74d8cffbe95552) I0917 09:19:50.039037 15404 raft_consensus.cc:494] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:50.039139 15404 leader_election.cc:296] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65:39971), eea10e830a664ea4be74d8cffbe95552 (127.13.208.67:42065) I0917 09:19:50.039384 15318 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f67125c2201a4791ab5756ecc8fba82a" candidate_uuid: "f42c56d2dc6b4faba5a218e79da90951" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "eea10e830a664ea4be74d8cffbe95552" is_pre_election: true I0917 09:19:50.039405 15166 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f67125c2201a4791ab5756ecc8fba82a" candidate_uuid: "f42c56d2dc6b4faba5a218e79da90951" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c2c8bec4d40141a6b1f470f6142de67a" is_pre_election: true I0917 09:19:50.039422 15318 raft_consensus.cc:2280] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f42c56d2dc6b4faba5a218e79da90951 in current term 2: Already voted for candidate c2c8bec4d40141a6b1f470f6142de67a in this term. I0917 09:19:50.039450 15166 raft_consensus.cc:2311] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate f42c56d2dc6b4faba5a218e79da90951 for term 2 because replica is either leader or believes a valid leader to be alive. I0917 09:19:50.040100 15318 tablet_service.cc:1440] Received LeaderStepDown RPC: tablet_id: "de49f5bae5244941af0020522f04d827" dest_uuid: "eea10e830a664ea4be74d8cffbe95552" from {username='slave'} at 127.0.0.1:58758 I0917 09:19:50.040191 15318 raft_consensus.cc:569] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [term 1 LEADER]: Received request to step down I0917 09:19:50.040211 15318 raft_consensus.cc:2885] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [term 1 LEADER]: Stepping down as leader of term 1 I0917 09:19:50.040232 15318 raft_consensus.cc:705] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eea10e830a664ea4be74d8cffbe95552, State: Running, Role: LEADER I0917 09:19:50.040452 15318 consensus_queue.cc:229] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:50.040526 15318 raft_consensus.cc:2890] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [term 1 FOLLOWER]: Advancing to term 2 I0917 09:19:50.040594 15202 leader_election.cc:310] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f42c56d2dc6b4faba5a218e79da90951; no voters: c2c8bec4d40141a6b1f470f6142de67a, eea10e830a664ea4be74d8cffbe95552 I0917 09:19:50.041244 15404 raft_consensus.cc:2890] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 1 FOLLOWER]: Advancing to term 2 I0917 09:19:50.041985 15404 raft_consensus.cc:2592] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0917 09:19:50.114351 15116 catalog_manager.cc:1142] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b: acquiring CA information for follower catalog manager: success I0917 09:19:50.114606 15116 catalog_manager.cc:1170] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0917 09:19:50.117048 15243 raft_consensus.cc:1184] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 2 FOLLOWER]: Refusing update from remote peer c2c8bec4d40141a6b1f470f6142de67a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch) I0917 09:19:50.117345 15408 consensus_queue.cc:984] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0917 09:19:50.118863 15318 raft_consensus.cc:1184] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 2 FOLLOWER]: Refusing update from remote peer c2c8bec4d40141a6b1f470f6142de67a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch) I0917 09:19:50.119158 15408 consensus_queue.cc:984] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0917 09:19:50.152415 15122 catalog_manager.cc:1142] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27: acquiring CA information for follower catalog manager: success I0917 09:19:50.152596 15122 catalog_manager.cc:1170] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0917 09:19:50.184114 15408 raft_consensus.cc:472] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eea10e830a664ea4be74d8cffbe95552) I0917 09:19:50.184232 15408 raft_consensus.cc:494] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:50.184334 15408 leader_election.cc:296] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793), eea10e830a664ea4be74d8cffbe95552 (127.13.208.67:42065) I0917 09:19:50.184574 15243 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "de49f5bae5244941af0020522f04d827" candidate_uuid: "c2c8bec4d40141a6b1f470f6142de67a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f42c56d2dc6b4faba5a218e79da90951" is_pre_election: true I0917 09:19:50.184626 15243 raft_consensus.cc:2311] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c2c8bec4d40141a6b1f470f6142de67a for term 2 because replica is either leader or believes a valid leader to be alive. I0917 09:19:50.184659 15318 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "de49f5bae5244941af0020522f04d827" candidate_uuid: "c2c8bec4d40141a6b1f470f6142de67a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "eea10e830a664ea4be74d8cffbe95552" is_pre_election: true I0917 09:19:50.184698 15318 raft_consensus.cc:2350] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2c8bec4d40141a6b1f470f6142de67a in term 2. I0917 09:19:50.184878 15128 leader_election.cc:310] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c2c8bec4d40141a6b1f470f6142de67a, eea10e830a664ea4be74d8cffbe95552; no voters: f42c56d2dc6b4faba5a218e79da90951 I0917 09:19:50.185007 15408 raft_consensus.cc:2644] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [term 1 FOLLOWER]: Leader pre-election won for term 2 I0917 09:19:50.185024 15408 raft_consensus.cc:472] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [term 1 FOLLOWER]: Starting leader election (detected failure of leader eea10e830a664ea4be74d8cffbe95552) I0917 09:19:50.185031 15408 raft_consensus.cc:2890] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [term 1 FOLLOWER]: Advancing to term 2 I0917 09:19:50.186030 15404 raft_consensus.cc:472] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eea10e830a664ea4be74d8cffbe95552) I0917 09:19:50.186187 15408 raft_consensus.cc:494] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:50.186192 15404 raft_consensus.cc:494] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:50.186308 15408 leader_election.cc:296] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [CANDIDATE]: Term 2 election: Requested vote from peers f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793), eea10e830a664ea4be74d8cffbe95552 (127.13.208.67:42065) I0917 09:19:50.186537 15166 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "de49f5bae5244941af0020522f04d827" candidate_uuid: "f42c56d2dc6b4faba5a218e79da90951" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c2c8bec4d40141a6b1f470f6142de67a" is_pre_election: true I0917 09:19:50.186543 15404 leader_election.cc:296] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65:39971), eea10e830a664ea4be74d8cffbe95552 (127.13.208.67:42065) I0917 09:19:50.186580 15318 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "de49f5bae5244941af0020522f04d827" candidate_uuid: "c2c8bec4d40141a6b1f470f6142de67a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "eea10e830a664ea4be74d8cffbe95552" I0917 09:19:50.186658 15317 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "de49f5bae5244941af0020522f04d827" candidate_uuid: "f42c56d2dc6b4faba5a218e79da90951" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "eea10e830a664ea4be74d8cffbe95552" is_pre_election: true I0917 09:19:50.186682 15166 raft_consensus.cc:2280] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f42c56d2dc6b4faba5a218e79da90951 in current term 2: Already voted for candidate c2c8bec4d40141a6b1f470f6142de67a in this term. I0917 09:19:50.186839 15243 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "de49f5bae5244941af0020522f04d827" candidate_uuid: "c2c8bec4d40141a6b1f470f6142de67a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f42c56d2dc6b4faba5a218e79da90951" I0917 09:19:50.186900 15243 raft_consensus.cc:2890] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [term 1 FOLLOWER]: Advancing to term 2 I0917 09:19:50.187870 15318 raft_consensus.cc:2350] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2c8bec4d40141a6b1f470f6142de67a in term 2. I0917 09:19:50.187912 15317 raft_consensus.cc:2325] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f42c56d2dc6b4faba5a218e79da90951 for term 2 because replica is already servicing an update from a current leader or another vote. I0917 09:19:50.188031 15128 leader_election.cc:310] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2c8bec4d40141a6b1f470f6142de67a, eea10e830a664ea4be74d8cffbe95552; no voters: I0917 09:19:50.188096 15408 raft_consensus.cc:2644] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [term 2 FOLLOWER]: Leader election won for term 2 I0917 09:19:50.188118 15408 raft_consensus.cc:667] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [term 2 LEADER]: Becoming Leader. State: Replica: c2c8bec4d40141a6b1f470f6142de67a, State: Running, Role: LEADER I0917 09:19:50.188150 15243 raft_consensus.cc:2350] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2c8bec4d40141a6b1f470f6142de67a in term 2. I0917 09:19:50.188170 15202 leader_election.cc:310] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f42c56d2dc6b4faba5a218e79da90951; no voters: c2c8bec4d40141a6b1f470f6142de67a, eea10e830a664ea4be74d8cffbe95552 I0917 09:19:50.188351 15404 raft_consensus.cc:2592] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0917 09:19:50.188380 15408 consensus_queue.cc:206] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:50.189069 14958 catalog_manager.cc:4120] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a reported cstate change: term changed from 1 to 2, leader changed from eea10e830a664ea4be74d8cffbe95552 (127.13.208.67) to c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65). New cstate: current_term: 2 leader_uuid: "c2c8bec4d40141a6b1f470f6142de67a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } health_report { overall_health: UNKNOWN } } } I0917 09:19:50.241948 14960 catalog_manager.cc:2365] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43034: table { table_id: "03b5e80e6ff342bab99657ab154c5e00" table_name: "test-table" } new_table_name: "NewTableName" I0917 09:19:50.244258 14145 master.cc:247] Master@127.13.208.126:33685 shutting down... I0917 09:19:50.245548 15241 raft_consensus.cc:1184] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [term 2 FOLLOWER]: Refusing update from remote peer c2c8bec4d40141a6b1f470f6142de67a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch) I0917 09:19:50.245570 15418 tablet.cc:1223] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: Alter schema from ( 0:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 0:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0917 09:19:50.246073 15317 raft_consensus.cc:1184] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [term 2 FOLLOWER]: Refusing update from remote peer c2c8bec4d40141a6b1f470f6142de67a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch) I0917 09:19:50.246129 15408 consensus_queue.cc:984] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0917 09:19:50.246153 15417 tablet.cc:1223] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552: Alter schema from ( 0:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 0:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0917 09:19:50.247246 14945 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0917 09:19:50.246445 15419 tablet.cc:1223] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a: Alter schema from ( 0:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 0:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0917 09:19:50.247531 15408 consensus_queue.cc:984] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0917 09:19:50.247802 15420 tablet.cc:1223] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552: Alter schema from ( 0:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 0:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0917 09:19:50.248051 15422 tablet.cc:1223] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951: Alter schema from ( 0:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 0:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0917 09:19:50.248509 14945 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0917 09:19:50.248579 15423 tablet.cc:1223] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951: Alter schema from ( 0:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 0:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0917 09:19:50.249439 15421 tablet.cc:1223] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a: Alter schema from ( 0:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 0:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0917 09:19:50.249658 14946 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0917 09:19:50.249886 15425 tablet.cc:1223] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552: Alter schema from ( 0:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 0:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0917 09:19:50.250671 15424 tablet.cc:1223] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a: Alter schema from ( 0:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 0:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0917 09:19:50.250846 14945 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0917 09:19:50.251317 14947 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0917 09:19:50.255858 14946 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0917 09:19:50.255884 14945 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0917 09:19:50.257203 14947 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0917 09:19:50.258205 14946 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0917 09:19:50.258589 14946 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0917 09:19:50.258651 14947 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0917 09:19:50.259199 14947 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0917 09:19:50.259423 14145 raft_consensus.cc:2147] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 1 LEADER]: Raft consensus shutting down. I0917 09:19:50.259605 14145 raft_consensus.cc:2174] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 1 FOLLOWER]: Raft consensus is shut down! I0917 09:19:50.259630 14145 tablet_replica.cc:273] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266: stopping tablet replica I0917 09:19:50.259658 14945 messenger.cc:381] Service unavailable: service kudu.master.MasterService not registered on Master I0917 09:19:50.271697 14145 master.cc:258] Master@127.13.208.126:33685 shutdown complete. I0917 09:19:50.444324 15429 raft_consensus.cc:472] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader abfd3dc05c4a4566bb328f04255eb266) I0917 09:19:50.444464 15429 raft_consensus.cc:494] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } I0917 09:19:50.444568 15429 leader_election.cc:296] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers abfd3dc05c4a4566bb328f04255eb266 (127.13.208.126:33685), e501d8ef778246b69b34d86e7395022b (127.13.208.124:39307) I0917 09:19:50.444604 15430 raft_consensus.cc:472] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader abfd3dc05c4a4566bb328f04255eb266) I0917 09:19:50.444748 15430 raft_consensus.cc:494] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } I0917 09:19:50.444941 15430 leader_election.cc:296] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers abfd3dc05c4a4566bb328f04255eb266 (127.13.208.126:33685), 82e2633778254c44a45393ac32540f27 (127.13.208.125:37503) I0917 09:19:50.445086 15030 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e501d8ef778246b69b34d86e7395022b" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "82e2633778254c44a45393ac32540f27" is_pre_election: true W0917 09:19:50.445102 15003 leader_election.cc:341] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer abfd3dc05c4a4566bb328f04255eb266 (127.13.208.126:33685): Network error: Client connection negotiation failed: client connection to 127.13.208.126:33685: connect: Connection refused (error 111) I0917 09:19:50.445133 15030 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e501d8ef778246b69b34d86e7395022b in term 1. I0917 09:19:50.445365 15058 leader_election.cc:310] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82e2633778254c44a45393ac32540f27, e501d8ef778246b69b34d86e7395022b; no voters: W0917 09:19:50.445436 15059 leader_election.cc:341] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer abfd3dc05c4a4566bb328f04255eb266 (127.13.208.126:33685): Network error: Client connection negotiation failed: client connection to 127.13.208.126:33685: connect: Connection refused (error 111) I0917 09:19:50.445451 15430 raft_consensus.cc:2644] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 1 FOLLOWER]: Leader pre-election won for term 2 I0917 09:19:50.445446 15088 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82e2633778254c44a45393ac32540f27" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "e501d8ef778246b69b34d86e7395022b" is_pre_election: true I0917 09:19:50.445472 15430 raft_consensus.cc:472] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 1 FOLLOWER]: Starting leader election (detected failure of leader abfd3dc05c4a4566bb328f04255eb266) I0917 09:19:50.445508 15430 raft_consensus.cc:2890] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 1 FOLLOWER]: Advancing to term 2 I0917 09:19:50.446368 15430 raft_consensus.cc:494] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } I0917 09:19:50.446458 15088 raft_consensus.cc:2280] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82e2633778254c44a45393ac32540f27 in current term 2: Already voted for candidate e501d8ef778246b69b34d86e7395022b in this term. I0917 09:19:50.446607 15430 leader_election.cc:296] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [CANDIDATE]: Term 2 election: Requested vote from peers abfd3dc05c4a4566bb328f04255eb266 (127.13.208.126:33685), 82e2633778254c44a45393ac32540f27 (127.13.208.125:37503) I0917 09:19:50.446703 15003 leader_election.cc:310] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82e2633778254c44a45393ac32540f27; no voters: abfd3dc05c4a4566bb328f04255eb266, e501d8ef778246b69b34d86e7395022b I0917 09:19:50.446794 15030 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e501d8ef778246b69b34d86e7395022b" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "82e2633778254c44a45393ac32540f27" I0917 09:19:50.446800 15429 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 1 FOLLOWER]: Advancing to term 2 W0917 09:19:50.447003 15059 leader_election.cc:341] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer abfd3dc05c4a4566bb328f04255eb266 (127.13.208.126:33685): Network error: Client connection negotiation failed: client connection to 127.13.208.126:33685: connect: Connection refused (error 111) I0917 09:19:50.447724 15429 raft_consensus.cc:2592] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0917 09:19:50.449074 15030 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e501d8ef778246b69b34d86e7395022b in term 2. I0917 09:19:50.449280 15058 leader_election.cc:310] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82e2633778254c44a45393ac32540f27, e501d8ef778246b69b34d86e7395022b; no voters: abfd3dc05c4a4566bb328f04255eb266 I0917 09:19:50.449365 15430 raft_consensus.cc:2644] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 2 FOLLOWER]: Leader election won for term 2 I0917 09:19:50.449486 15430 raft_consensus.cc:667] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 2 LEADER]: Becoming Leader. State: Replica: e501d8ef778246b69b34d86e7395022b, State: Running, Role: LEADER I0917 09:19:50.449581 15430 consensus_queue.cc:206] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } I0917 09:19:50.449818 15433 sys_catalog.cc:340] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e501d8ef778246b69b34d86e7395022b. Latest consensus state: current_term: 2 leader_uuid: "e501d8ef778246b69b34d86e7395022b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:50.449842 15433 sys_catalog.cc:343] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [sys.catalog]: This master's current role is: LEADER I0917 09:19:50.449990 15434 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0917 09:19:50.450248 15434 catalog_manager.cc:387] Loaded metadata for table NewTableName [id=03b5e80e6ff342bab99657ab154c5e00] I0917 09:19:50.450407 15434 catalog_manager.cc:441] Loaded metadata for tablet ac74b319ad54416685f8b9d9506e1d61 (table NewTableName [id=03b5e80e6ff342bab99657ab154c5e00]) I0917 09:19:50.450438 15434 catalog_manager.cc:441] Loaded metadata for tablet de49f5bae5244941af0020522f04d827 (table NewTableName [id=03b5e80e6ff342bab99657ab154c5e00]) I0917 09:19:50.450457 15434 catalog_manager.cc:441] Loaded metadata for tablet f67125c2201a4791ab5756ecc8fba82a (table NewTableName [id=03b5e80e6ff342bab99657ab154c5e00]) I0917 09:19:50.450479 15434 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0917 09:19:50.450786 15434 catalog_manager.cc:1106] Loading token signing keys... I0917 09:19:50.450863 15434 catalog_manager.cc:4446] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b: Loaded TSK: 0 I0917 09:19:50.477110 14145 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 27804us I0917 09:19:50.477198 14145 fs_manager.cc:263] Metadata directory not provided I0917 09:19:50.477216 14145 fs_manager.cc:266] Using existing metadata directory in first data directory I0917 09:19:50.477793 14145 fs_manager.cc:399] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s I0917 09:19:50.478030 14145 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0917 09:19:50.480010 14145 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:50.480082 14145 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/master-0-root uuid: "abfd3dc05c4a4566bb328f04255eb266" format_stamp: "Formatted at 2019-09-17 09:19:49 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:50.480101 14145 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/master-0-root metadata directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0917 09:19:50.509876 14145 rpc_server.cc:205] RPC server started. Bound to: 127.13.208.126:33685 I0917 09:19:50.510192 14145 webserver.cc:374] Webserver started at http://127.13.208.126:44757/ using document root and password file I0917 09:19:50.511214 15489 sys_catalog.cc:188] Verifying existing consensus state I0917 09:19:50.511653 15489 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266: Bootstrap starting. I0917 09:19:50.513546 15489 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=6 ignored=0} mutations{seen=11 ignored=0} orphaned_commits=0. Pending: 0 replicates I0917 09:19:50.513793 15489 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266: Bootstrap complete. I0917 09:19:50.514014 15489 raft_consensus.cc:340] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } I0917 09:19:50.514032 15489 raft_consensus.cc:705] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: abfd3dc05c4a4566bb328f04255eb266, State: Initialized, Role: FOLLOWER I0917 09:19:50.514097 15489 consensus_queue.cc:229] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } I0917 09:19:50.514274 15489 sys_catalog.cc:340] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:50.514304 15491 sys_catalog.cc:340] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:50.514322 15489 sys_catalog.cc:343] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [sys.catalog]: This master's current role is: FOLLOWER I0917 09:19:50.514340 15491 sys_catalog.cc:343] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [sys.catalog]: This master's current role is: FOLLOWER I0917 09:19:50.514391 15489 sys_catalog.cc:428] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [sys.catalog]: configured and running, proceeding with master startup. I0917 09:19:50.515398 15492 catalog_manager.cc:1142] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266: acquiring CA information for follower catalog manager: success I0917 09:19:50.515532 15492 catalog_manager.cc:1170] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0917 09:19:50.543475 15030 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 2 FOLLOWER]: Refusing update from remote peer e501d8ef778246b69b34d86e7395022b: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch) I0917 09:19:50.543803 15433 consensus_queue.cc:984] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [LEADER]: Connected to new peer: Peer: permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s I0917 09:19:50.544114 15429 sys_catalog.cc:340] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e501d8ef778246b69b34d86e7395022b. Latest consensus state: current_term: 2 leader_uuid: "e501d8ef778246b69b34d86e7395022b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:50.544140 15429 sys_catalog.cc:343] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [sys.catalog]: This master's current role is: FOLLOWER I0917 09:19:50.544530 15430 sys_catalog.cc:340] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "e501d8ef778246b69b34d86e7395022b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:50.544560 15430 sys_catalog.cc:343] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [sys.catalog]: This master's current role is: LEADER I0917 09:19:50.544688 15429 sys_catalog.cc:340] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "e501d8ef778246b69b34d86e7395022b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:50.544714 15429 sys_catalog.cc:343] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [sys.catalog]: This master's current role is: FOLLOWER I0917 09:19:50.547726 15466 raft_consensus.cc:2890] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 1 FOLLOWER]: Advancing to term 2 I0917 09:19:50.548527 15466 raft_consensus.cc:1184] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 2 FOLLOWER]: Refusing update from remote peer e501d8ef778246b69b34d86e7395022b: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch) I0917 09:19:50.548780 15433 consensus_queue.cc:984] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [LEADER]: Connected to new peer: Peer: permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s I0917 09:19:50.549366 15496 sys_catalog.cc:340] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "e501d8ef778246b69b34d86e7395022b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:50.549391 15496 sys_catalog.cc:343] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [sys.catalog]: This master's current role is: FOLLOWER I0917 09:19:50.549412 15491 sys_catalog.cc:340] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e501d8ef778246b69b34d86e7395022b. Latest consensus state: current_term: 2 leader_uuid: "e501d8ef778246b69b34d86e7395022b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:50.549443 15491 sys_catalog.cc:343] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [sys.catalog]: This master's current role is: FOLLOWER I0917 09:19:50.561635 15269 heartbeater.cc:447] Master 127.13.208.124:39307 was elected leader, sending a full tablet report... I0917 09:19:50.561815 15341 heartbeater.cc:447] Master 127.13.208.124:39307 was elected leader, sending a full tablet report... I0917 09:19:50.563385 15430 sys_catalog.cc:340] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "e501d8ef778246b69b34d86e7395022b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:50.563414 15430 sys_catalog.cc:343] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [sys.catalog]: This master's current role is: LEADER I0917 09:19:50.563740 15077 catalog_manager.cc:4580] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b: NewTableName [id=03b5e80e6ff342bab99657ab154c5e00] alter complete (version 1) I0917 09:19:50.565927 15271 heartbeater.cc:322] Connected to a master server at 127.13.208.126:33685 I0917 09:19:50.566304 15456 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" instance_seqno: 1568711989250571) as {username='slave'} at 127.0.0.1:43068; Asking this server to re-register. I0917 09:19:50.566457 15271 heartbeater.cc:406] Registering TS with master... I0917 09:19:50.566855 15456 ts_manager.cc:161] Registered new tserver with Master: f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793) I0917 09:19:50.567029 15343 heartbeater.cc:322] Connected to a master server at 127.13.208.126:33685 I0917 09:19:50.567307 15456 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" instance_seqno: 1568711989299382) as {username='slave'} at 127.0.0.1:43070; Asking this server to re-register. I0917 09:19:50.567450 15343 heartbeater.cc:406] Registering TS with master... I0917 09:19:50.567711 15190 heartbeater.cc:447] Master 127.13.208.124:39307 was elected leader, sending a full tablet report... I0917 09:19:50.567816 15456 ts_manager.cc:161] Registered new tserver with Master: eea10e830a664ea4be74d8cffbe95552 (127.13.208.67:42065) I0917 09:19:50.570487 15192 heartbeater.cc:322] Connected to a master server at 127.13.208.126:33685 I0917 09:19:50.570816 15456 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" instance_seqno: 1568711989214369) as {username='slave'} at 127.0.0.1:43072; Asking this server to re-register. I0917 09:19:50.570994 15192 heartbeater.cc:406] Registering TS with master... I0917 09:19:50.571240 15456 ts_manager.cc:161] Registered new tserver with Master: c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65:39971) I0917 09:19:50.613472 14145 master.cc:247] Master@127.13.208.124:39307 shutting down... I0917 09:19:50.616817 14145 raft_consensus.cc:2147] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 2 LEADER]: Raft consensus shutting down. I0917 09:19:50.616937 14145 raft_consensus.cc:2174] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 2 FOLLOWER]: Raft consensus is shut down! I0917 09:19:50.616993 14145 tablet_replica.cc:273] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b: stopping tablet replica I0917 09:19:50.629048 14145 master.cc:258] Master@127.13.208.124:39307 shutdown complete. I0917 09:19:50.764243 15429 raft_consensus.cc:472] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e501d8ef778246b69b34d86e7395022b) I0917 09:19:50.764371 15429 raft_consensus.cc:494] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } I0917 09:19:50.764472 15491 raft_consensus.cc:472] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e501d8ef778246b69b34d86e7395022b) I0917 09:19:50.764546 15429 leader_election.cc:296] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers abfd3dc05c4a4566bb328f04255eb266 (127.13.208.126:33685), e501d8ef778246b69b34d86e7395022b (127.13.208.124:39307) I0917 09:19:50.764618 15491 raft_consensus.cc:494] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } I0917 09:19:50.764765 15491 leader_election.cc:296] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82e2633778254c44a45393ac32540f27 (127.13.208.125:37503), e501d8ef778246b69b34d86e7395022b (127.13.208.124:39307) W0917 09:19:50.765203 15438 leader_election.cc:341] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e501d8ef778246b69b34d86e7395022b (127.13.208.124:39307): Network error: Client connection negotiation failed: client connection to 127.13.208.124:39307: connect: Connection refused (error 111) W0917 09:19:50.765214 15003 leader_election.cc:341] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e501d8ef778246b69b34d86e7395022b (127.13.208.124:39307): Network error: Client connection negotiation failed: client connection to 127.13.208.124:39307: connect: Connection refused (error 111) I0917 09:19:50.769199 15030 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "abfd3dc05c4a4566bb328f04255eb266" candidate_term: 3 candidate_status { last_received { term: 2 index: 15 } } ignore_live_leader: false dest_uuid: "82e2633778254c44a45393ac32540f27" is_pre_election: true I0917 09:19:50.769204 15465 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82e2633778254c44a45393ac32540f27" candidate_term: 3 candidate_status { last_received { term: 2 index: 15 } } ignore_live_leader: false dest_uuid: "abfd3dc05c4a4566bb328f04255eb266" is_pre_election: true I0917 09:19:50.769253 15030 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abfd3dc05c4a4566bb328f04255eb266 in term 2. I0917 09:19:50.769256 15465 raft_consensus.cc:2350] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82e2633778254c44a45393ac32540f27 in term 2. I0917 09:19:50.769367 15003 leader_election.cc:310] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82e2633778254c44a45393ac32540f27, abfd3dc05c4a4566bb328f04255eb266; no voters: e501d8ef778246b69b34d86e7395022b I0917 09:19:50.769372 15437 leader_election.cc:310] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82e2633778254c44a45393ac32540f27, abfd3dc05c4a4566bb328f04255eb266; no voters: e501d8ef778246b69b34d86e7395022b I0917 09:19:50.769467 15491 raft_consensus.cc:2644] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 2 FOLLOWER]: Leader pre-election won for term 3 I0917 09:19:50.769474 15429 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 2 FOLLOWER]: Leader pre-election won for term 3 I0917 09:19:50.769484 15491 raft_consensus.cc:472] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e501d8ef778246b69b34d86e7395022b) I0917 09:19:50.769493 15429 raft_consensus.cc:472] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e501d8ef778246b69b34d86e7395022b) I0917 09:19:50.769498 15491 raft_consensus.cc:2890] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 2 FOLLOWER]: Advancing to term 3 I0917 09:19:50.769505 15429 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 2 FOLLOWER]: Advancing to term 3 I0917 09:19:50.771342 15491 raft_consensus.cc:494] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } I0917 09:19:50.771466 15491 leader_election.cc:296] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [CANDIDATE]: Term 3 election: Requested vote from peers 82e2633778254c44a45393ac32540f27 (127.13.208.125:37503), e501d8ef778246b69b34d86e7395022b (127.13.208.124:39307) I0917 09:19:50.771476 15429 raft_consensus.cc:494] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } I0917 09:19:50.771715 15030 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "abfd3dc05c4a4566bb328f04255eb266" candidate_term: 3 candidate_status { last_received { term: 2 index: 15 } } ignore_live_leader: false dest_uuid: "82e2633778254c44a45393ac32540f27" I0917 09:19:50.771747 15030 raft_consensus.cc:2280] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate abfd3dc05c4a4566bb328f04255eb266 in current term 3: Already voted for candidate 82e2633778254c44a45393ac32540f27 in this term. W0917 09:19:50.771947 15438 leader_election.cc:341] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e501d8ef778246b69b34d86e7395022b (127.13.208.124:39307): Network error: Client connection negotiation failed: client connection to 127.13.208.124:39307: connect: Connection refused (error 111) I0917 09:19:50.771980 15438 leader_election.cc:310] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: abfd3dc05c4a4566bb328f04255eb266; no voters: 82e2633778254c44a45393ac32540f27, e501d8ef778246b69b34d86e7395022b I0917 09:19:50.772029 15491 raft_consensus.cc:2592] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority I0917 09:19:50.773028 15429 leader_election.cc:296] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [CANDIDATE]: Term 3 election: Requested vote from peers abfd3dc05c4a4566bb328f04255eb266 (127.13.208.126:33685), e501d8ef778246b69b34d86e7395022b (127.13.208.124:39307) W0917 09:19:50.773223 15003 leader_election.cc:341] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e501d8ef778246b69b34d86e7395022b (127.13.208.124:39307): Network error: Client connection negotiation failed: client connection to 127.13.208.124:39307: connect: Connection refused (error 111) I0917 09:19:50.773301 15465 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82e2633778254c44a45393ac32540f27" candidate_term: 3 candidate_status { last_received { term: 2 index: 15 } } ignore_live_leader: false dest_uuid: "abfd3dc05c4a4566bb328f04255eb266" I0917 09:19:50.773332 15465 raft_consensus.cc:2280] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 82e2633778254c44a45393ac32540f27 in current term 3: Already voted for candidate abfd3dc05c4a4566bb328f04255eb266 in this term. I0917 09:19:50.773485 15003 leader_election.cc:310] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82e2633778254c44a45393ac32540f27; no voters: abfd3dc05c4a4566bb328f04255eb266, e501d8ef778246b69b34d86e7395022b I0917 09:19:50.773583 15429 raft_consensus.cc:2592] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority I0917 09:19:50.833788 14145 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 27804us I0917 09:19:50.833861 14145 fs_manager.cc:263] Metadata directory not provided I0917 09:19:50.833878 14145 fs_manager.cc:266] Using existing metadata directory in first data directory I0917 09:19:50.837272 14145 fs_manager.cc:399] Time spent opening directory manager: real 0.003s user 0.000s sys 0.001s I0917 09:19:50.837605 14145 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0917 09:19:50.839455 14145 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:50.839506 14145 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/master-2-root uuid: "e501d8ef778246b69b34d86e7395022b" format_stamp: "Formatted at 2019-09-17 09:19:49 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:50.839529 14145 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/master-2-root metadata directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/master-2-root 1 data directories: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/master-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0917 09:19:50.863963 14145 rpc_server.cc:205] RPC server started. Bound to: 127.13.208.124:39307 I0917 09:19:50.864280 14145 webserver.cc:374] Webserver started at http://127.13.208.124:44755/ using document root and password file I0917 09:19:50.865206 15557 sys_catalog.cc:188] Verifying existing consensus state I0917 09:19:50.865568 15557 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b: Bootstrap starting. I0917 09:19:50.867156 15557 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=0} inserts{seen=6 ignored=0} mutations{seen=12 ignored=0} orphaned_commits=0. Pending: 0 replicates I0917 09:19:50.867360 15557 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b: Bootstrap complete. I0917 09:19:50.867552 15557 raft_consensus.cc:340] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } I0917 09:19:50.867571 15557 raft_consensus.cc:705] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e501d8ef778246b69b34d86e7395022b, State: Initialized, Role: FOLLOWER I0917 09:19:50.867640 15557 consensus_queue.cc:229] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 2.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } I0917 09:19:50.867753 15557 sys_catalog.cc:340] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:50.867781 15557 sys_catalog.cc:343] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [sys.catalog]: This master's current role is: FOLLOWER I0917 09:19:50.867882 15557 sys_catalog.cc:428] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [sys.catalog]: configured and running, proceeding with master startup. I0917 09:19:50.867918 15559 sys_catalog.cc:340] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:50.867942 15559 sys_catalog.cc:343] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [sys.catalog]: This master's current role is: FOLLOWER I0917 09:19:50.869392 15560 catalog_manager.cc:1142] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b: acquiring CA information for follower catalog manager: success I0917 09:19:50.869572 15560 catalog_manager.cc:1170] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0917 09:19:50.917783 15269 heartbeater.cc:322] Connected to a master server at 127.13.208.124:39307 I0917 09:19:50.918251 15522 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" instance_seqno: 1568711989250571) as {username='slave'} at 127.0.0.1:58094; Asking this server to re-register. I0917 09:19:50.918378 15269 heartbeater.cc:406] Registering TS with master... I0917 09:19:50.918805 15522 ts_manager.cc:161] Registered new tserver with Master: f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793) I0917 09:19:50.919644 15341 heartbeater.cc:322] Connected to a master server at 127.13.208.124:39307 I0917 09:19:50.919720 15190 heartbeater.cc:322] Connected to a master server at 127.13.208.124:39307 I0917 09:19:50.919875 15522 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" instance_seqno: 1568711989299382) as {username='slave'} at 127.0.0.1:58096; Asking this server to re-register. I0917 09:19:50.919975 15524 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" instance_seqno: 1568711989214369) as {username='slave'} at 127.0.0.1:58098; Asking this server to re-register. I0917 09:19:50.920001 15341 heartbeater.cc:406] Registering TS with master... I0917 09:19:50.920107 15190 heartbeater.cc:406] Registering TS with master... I0917 09:19:50.920228 15524 ts_manager.cc:161] Registered new tserver with Master: eea10e830a664ea4be74d8cffbe95552 (127.13.208.67:42065) I0917 09:19:50.920356 15522 ts_manager.cc:161] Registered new tserver with Master: c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65:39971) I0917 09:19:51.046635 15429 raft_consensus.cc:472] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0917 09:19:51.046795 15429 raft_consensus.cc:494] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } I0917 09:19:51.047214 15429 leader_election.cc:296] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers abfd3dc05c4a4566bb328f04255eb266 (127.13.208.126:33685), e501d8ef778246b69b34d86e7395022b (127.13.208.124:39307) I0917 09:19:51.047574 15465 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82e2633778254c44a45393ac32540f27" candidate_term: 4 candidate_status { last_received { term: 2 index: 15 } } ignore_live_leader: false dest_uuid: "abfd3dc05c4a4566bb328f04255eb266" is_pre_election: true I0917 09:19:51.047677 15465 raft_consensus.cc:2350] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82e2633778254c44a45393ac32540f27 in term 3. I0917 09:19:51.047808 15003 leader_election.cc:310] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82e2633778254c44a45393ac32540f27, abfd3dc05c4a4566bb328f04255eb266; no voters: I0917 09:19:51.047940 15429 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 3 FOLLOWER]: Leader pre-election won for term 4 I0917 09:19:51.047969 15429 raft_consensus.cc:472] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0917 09:19:51.047978 15429 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 3 FOLLOWER]: Advancing to term 4 I0917 09:19:51.050913 15525 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82e2633778254c44a45393ac32540f27" candidate_term: 4 candidate_status { last_received { term: 2 index: 15 } } ignore_live_leader: false dest_uuid: "e501d8ef778246b69b34d86e7395022b" is_pre_election: true I0917 09:19:51.050952 15525 raft_consensus.cc:2350] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82e2633778254c44a45393ac32540f27 in term 2. I0917 09:19:51.057184 15429 raft_consensus.cc:494] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } I0917 09:19:51.057297 15429 leader_election.cc:296] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [CANDIDATE]: Term 4 election: Requested vote from peers abfd3dc05c4a4566bb328f04255eb266 (127.13.208.126:33685), e501d8ef778246b69b34d86e7395022b (127.13.208.124:39307) I0917 09:19:51.057533 15465 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82e2633778254c44a45393ac32540f27" candidate_term: 4 candidate_status { last_received { term: 2 index: 15 } } ignore_live_leader: false dest_uuid: "abfd3dc05c4a4566bb328f04255eb266" I0917 09:19:51.057570 15465 raft_consensus.cc:2890] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 3 FOLLOWER]: Advancing to term 4 I0917 09:19:51.057582 15525 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82e2633778254c44a45393ac32540f27" candidate_term: 4 candidate_status { last_received { term: 2 index: 15 } } ignore_live_leader: false dest_uuid: "e501d8ef778246b69b34d86e7395022b" I0917 09:19:51.057687 15525 raft_consensus.cc:2890] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 2 FOLLOWER]: Advancing to term 4 I0917 09:19:51.059206 15465 raft_consensus.cc:2350] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82e2633778254c44a45393ac32540f27 in term 4. I0917 09:19:51.059247 15525 raft_consensus.cc:2350] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82e2633778254c44a45393ac32540f27 in term 4. I0917 09:19:51.059387 15003 leader_election.cc:310] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82e2633778254c44a45393ac32540f27, e501d8ef778246b69b34d86e7395022b; no voters: I0917 09:19:51.059471 15429 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 4 FOLLOWER]: Leader election won for term 4 I0917 09:19:51.059551 15429 raft_consensus.cc:667] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 4 LEADER]: Becoming Leader. State: Replica: 82e2633778254c44a45393ac32540f27, State: Running, Role: LEADER I0917 09:19:51.059633 15429 consensus_queue.cc:206] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 2.15, Last appended by leader: 15, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } I0917 09:19:51.059815 15564 sys_catalog.cc:340] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82e2633778254c44a45393ac32540f27. Latest consensus state: current_term: 4 leader_uuid: "82e2633778254c44a45393ac32540f27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:51.059846 15564 sys_catalog.cc:343] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [sys.catalog]: This master's current role is: LEADER I0917 09:19:51.059950 15565 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0917 09:19:51.060125 15565 catalog_manager.cc:387] Loaded metadata for table NewTableName [id=03b5e80e6ff342bab99657ab154c5e00] I0917 09:19:51.060206 15565 catalog_manager.cc:441] Loaded metadata for tablet ac74b319ad54416685f8b9d9506e1d61 (table NewTableName [id=03b5e80e6ff342bab99657ab154c5e00]) I0917 09:19:51.060235 15565 catalog_manager.cc:441] Loaded metadata for tablet de49f5bae5244941af0020522f04d827 (table NewTableName [id=03b5e80e6ff342bab99657ab154c5e00]) I0917 09:19:51.060258 15565 catalog_manager.cc:441] Loaded metadata for tablet f67125c2201a4791ab5756ecc8fba82a (table NewTableName [id=03b5e80e6ff342bab99657ab154c5e00]) I0917 09:19:51.060277 15565 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0917 09:19:51.060459 15565 catalog_manager.cc:1106] Loading token signing keys... I0917 09:19:51.060510 15565 catalog_manager.cc:4446] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27: Loaded TSK: 0 I0917 09:19:51.147629 15465 raft_consensus.cc:1184] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 4 FOLLOWER]: Refusing update from remote peer 82e2633778254c44a45393ac32540f27: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 4 index: 16. (index mismatch) I0917 09:19:51.147917 15564 consensus_queue.cc:984] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s I0917 09:19:51.148272 15491 sys_catalog.cc:340] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82e2633778254c44a45393ac32540f27. Latest consensus state: current_term: 4 leader_uuid: "82e2633778254c44a45393ac32540f27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:51.148298 15491 sys_catalog.cc:343] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [sys.catalog]: This master's current role is: FOLLOWER I0917 09:19:51.148773 15429 sys_catalog.cc:340] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "82e2633778254c44a45393ac32540f27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:51.148802 15429 sys_catalog.cc:343] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [sys.catalog]: This master's current role is: LEADER I0917 09:19:51.148914 15491 sys_catalog.cc:340] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "82e2633778254c44a45393ac32540f27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:51.148931 15491 sys_catalog.cc:343] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [sys.catalog]: This master's current role is: FOLLOWER I0917 09:19:51.149049 15525 raft_consensus.cc:1184] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 4 FOLLOWER]: Refusing update from remote peer 82e2633778254c44a45393ac32540f27: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 4 index: 16. (index mismatch) I0917 09:19:51.149250 15429 consensus_queue.cc:984] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s I0917 09:19:51.149775 15559 sys_catalog.cc:340] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82e2633778254c44a45393ac32540f27. Latest consensus state: current_term: 4 leader_uuid: "82e2633778254c44a45393ac32540f27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:51.149860 15559 sys_catalog.cc:343] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [sys.catalog]: This master's current role is: FOLLOWER I0917 09:19:51.149919 15571 sys_catalog.cc:340] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "82e2633778254c44a45393ac32540f27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:51.149940 15571 sys_catalog.cc:343] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [sys.catalog]: This master's current role is: FOLLOWER I0917 09:19:51.166380 15191 heartbeater.cc:447] Master 127.13.208.125:37503 was elected leader, sending a full tablet report... I0917 09:19:51.166954 15270 heartbeater.cc:447] Master 127.13.208.125:37503 was elected leader, sending a full tablet report... I0917 09:19:51.167160 15342 heartbeater.cc:447] Master 127.13.208.125:37503 was elected leader, sending a full tablet report... I0917 09:19:51.195304 14145 master.cc:247] Master@127.13.208.125:37503 shutting down... I0917 09:19:51.198906 14145 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 4 LEADER]: Raft consensus shutting down. I0917 09:19:51.198995 14145 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 4 FOLLOWER]: Raft consensus is shut down! I0917 09:19:51.199067 14145 tablet_replica.cc:273] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27: stopping tablet replica I0917 09:19:51.211019 14145 master.cc:258] Master@127.13.208.125:37503 shutdown complete. I0917 09:19:51.349858 15491 raft_consensus.cc:472] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 82e2633778254c44a45393ac32540f27) I0917 09:19:51.349980 15491 raft_consensus.cc:494] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } I0917 09:19:51.350072 15491 leader_election.cc:296] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82e2633778254c44a45393ac32540f27 (127.13.208.125:37503), e501d8ef778246b69b34d86e7395022b (127.13.208.124:39307) I0917 09:19:51.350425 15571 raft_consensus.cc:472] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 82e2633778254c44a45393ac32540f27) W0917 09:19:51.350512 15437 leader_election.cc:341] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 82e2633778254c44a45393ac32540f27 (127.13.208.125:37503): Network error: Client connection negotiation failed: client connection to 127.13.208.125:37503: connect: Connection refused (error 111) I0917 09:19:51.350569 15571 raft_consensus.cc:494] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } I0917 09:19:51.350953 15571 leader_election.cc:296] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers abfd3dc05c4a4566bb328f04255eb266 (127.13.208.126:33685), 82e2633778254c44a45393ac32540f27 (127.13.208.125:37503) W0917 09:19:51.351058 15505 leader_election.cc:341] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 82e2633778254c44a45393ac32540f27 (127.13.208.125:37503): Network error: Client connection negotiation failed: client connection to 127.13.208.125:37503: connect: Connection refused (error 111) I0917 09:19:51.354537 15525 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "abfd3dc05c4a4566bb328f04255eb266" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "e501d8ef778246b69b34d86e7395022b" is_pre_election: true I0917 09:19:51.354581 15525 raft_consensus.cc:2350] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abfd3dc05c4a4566bb328f04255eb266 in term 4. I0917 09:19:51.354709 15438 leader_election.cc:310] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: abfd3dc05c4a4566bb328f04255eb266, e501d8ef778246b69b34d86e7395022b; no voters: 82e2633778254c44a45393ac32540f27 I0917 09:19:51.354825 15491 raft_consensus.cc:2644] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 4 FOLLOWER]: Leader pre-election won for term 5 I0917 09:19:51.354857 15491 raft_consensus.cc:472] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 82e2633778254c44a45393ac32540f27) I0917 09:19:51.354871 15491 raft_consensus.cc:2890] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 4 FOLLOWER]: Advancing to term 5 I0917 09:19:51.355818 15465 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e501d8ef778246b69b34d86e7395022b" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "abfd3dc05c4a4566bb328f04255eb266" is_pre_election: true I0917 09:19:51.372725 15491 raft_consensus.cc:494] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } I0917 09:19:51.372939 15465 raft_consensus.cc:2280] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e501d8ef778246b69b34d86e7395022b in current term 5: Already voted for candidate abfd3dc05c4a4566bb328f04255eb266 in this term. I0917 09:19:51.372977 15491 leader_election.cc:296] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [CANDIDATE]: Term 5 election: Requested vote from peers 82e2633778254c44a45393ac32540f27 (127.13.208.125:37503), e501d8ef778246b69b34d86e7395022b (127.13.208.124:39307) I0917 09:19:51.373198 15525 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "abfd3dc05c4a4566bb328f04255eb266" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "e501d8ef778246b69b34d86e7395022b" I0917 09:19:51.373231 15525 raft_consensus.cc:2890] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 4 FOLLOWER]: Advancing to term 5 I0917 09:19:51.373324 15506 leader_election.cc:310] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e501d8ef778246b69b34d86e7395022b; no voters: 82e2633778254c44a45393ac32540f27, abfd3dc05c4a4566bb328f04255eb266 W0917 09:19:51.373555 15437 leader_election.cc:341] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 82e2633778254c44a45393ac32540f27 (127.13.208.125:37503): Network error: Client connection negotiation failed: client connection to 127.13.208.125:37503: connect: Connection refused (error 111) I0917 09:19:51.374526 15525 raft_consensus.cc:2350] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abfd3dc05c4a4566bb328f04255eb266 in term 5. I0917 09:19:51.374655 15571 raft_consensus.cc:2592] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I0917 09:19:51.374666 15438 leader_election.cc:310] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: abfd3dc05c4a4566bb328f04255eb266, e501d8ef778246b69b34d86e7395022b; no voters: 82e2633778254c44a45393ac32540f27 I0917 09:19:51.374747 15491 raft_consensus.cc:2644] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 5 FOLLOWER]: Leader election won for term 5 I0917 09:19:51.374831 15491 raft_consensus.cc:667] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 5 LEADER]: Becoming Leader. State: Replica: abfd3dc05c4a4566bb328f04255eb266, State: Running, Role: LEADER I0917 09:19:51.374904 15491 consensus_queue.cc:206] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } I0917 09:19:51.375178 15576 sys_catalog.cc:340] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abfd3dc05c4a4566bb328f04255eb266. Latest consensus state: current_term: 5 leader_uuid: "abfd3dc05c4a4566bb328f04255eb266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:51.375201 15576 sys_catalog.cc:343] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [sys.catalog]: This master's current role is: LEADER I0917 09:19:51.375291 15577 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0917 09:19:51.375491 15577 catalog_manager.cc:387] Loaded metadata for table NewTableName [id=03b5e80e6ff342bab99657ab154c5e00] I0917 09:19:51.375587 15577 catalog_manager.cc:441] Loaded metadata for tablet ac74b319ad54416685f8b9d9506e1d61 (table NewTableName [id=03b5e80e6ff342bab99657ab154c5e00]) I0917 09:19:51.375617 15577 catalog_manager.cc:441] Loaded metadata for tablet de49f5bae5244941af0020522f04d827 (table NewTableName [id=03b5e80e6ff342bab99657ab154c5e00]) I0917 09:19:51.375636 15577 catalog_manager.cc:441] Loaded metadata for tablet f67125c2201a4791ab5756ecc8fba82a (table NewTableName [id=03b5e80e6ff342bab99657ab154c5e00]) I0917 09:19:51.375658 15577 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0917 09:19:51.375833 15577 catalog_manager.cc:1106] Loading token signing keys... I0917 09:19:51.375897 15577 catalog_manager.cc:4446] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266: Loaded TSK: 0 I0917 09:19:51.415518 14145 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 28304us I0917 09:19:51.415597 14145 fs_manager.cc:263] Metadata directory not provided I0917 09:19:51.415612 14145 fs_manager.cc:266] Using existing metadata directory in first data directory I0917 09:19:51.416131 14145 fs_manager.cc:399] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:51.416359 14145 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0917 09:19:51.418233 14145 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:51.418323 14145 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/master-1-root uuid: "82e2633778254c44a45393ac32540f27" format_stamp: "Formatted at 2019-09-17 09:19:49 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:51.418339 14145 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/master-1-root metadata directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/master-1-root 1 data directories: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/master-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0917 09:19:51.437685 14145 rpc_server.cc:205] RPC server started. Bound to: 127.13.208.125:37503 I0917 09:19:51.438019 14145 webserver.cc:374] Webserver started at http://127.13.208.125:42235/ using document root and password file I0917 09:19:51.438971 15632 sys_catalog.cc:188] Verifying existing consensus state I0917 09:19:51.439615 15632 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27: Bootstrap starting. I0917 09:19:51.441483 15632 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=6 ignored=0} mutations{seen=12 ignored=0} orphaned_commits=0. Pending: 0 replicates I0917 09:19:51.441695 15632 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27: Bootstrap complete. I0917 09:19:51.441918 15632 raft_consensus.cc:340] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } I0917 09:19:51.441939 15632 raft_consensus.cc:705] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82e2633778254c44a45393ac32540f27, State: Initialized, Role: FOLLOWER I0917 09:19:51.442005 15632 consensus_queue.cc:229] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } I0917 09:19:51.442207 15632 sys_catalog.cc:340] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:51.442225 15632 sys_catalog.cc:343] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [sys.catalog]: This master's current role is: FOLLOWER I0917 09:19:51.442349 15634 sys_catalog.cc:340] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:51.442374 15632 sys_catalog.cc:428] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [sys.catalog]: configured and running, proceeding with master startup. I0917 09:19:51.442385 15634 sys_catalog.cc:343] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [sys.catalog]: This master's current role is: FOLLOWER I0917 09:19:51.443397 15635 catalog_manager.cc:1142] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27: acquiring CA information for follower catalog manager: success I0917 09:19:51.443540 15635 catalog_manager.cc:1170] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0917 09:19:51.459103 15609 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 4 FOLLOWER]: Advancing to term 5 I0917 09:19:51.460031 15609 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 5 FOLLOWER]: Refusing update from remote peer abfd3dc05c4a4566bb328f04255eb266: Log matching property violated. Preceding OpId in replica: term: 4 index: 16. Preceding OpId from leader: term: 5 index: 17. (index mismatch) I0917 09:19:51.460252 15576 consensus_queue.cc:984] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s I0917 09:19:51.460640 15634 sys_catalog.cc:340] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abfd3dc05c4a4566bb328f04255eb266. Latest consensus state: current_term: 5 leader_uuid: "abfd3dc05c4a4566bb328f04255eb266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:51.460665 15634 sys_catalog.cc:343] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [sys.catalog]: This master's current role is: FOLLOWER I0917 09:19:51.461086 15491 sys_catalog.cc:340] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "abfd3dc05c4a4566bb328f04255eb266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:51.461112 15491 sys_catalog.cc:343] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [sys.catalog]: This master's current role is: LEADER I0917 09:19:51.461302 15525 raft_consensus.cc:1184] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 5 FOLLOWER]: Refusing update from remote peer abfd3dc05c4a4566bb328f04255eb266: Log matching property violated. Preceding OpId in replica: term: 4 index: 16. Preceding OpId from leader: term: 5 index: 17. (index mismatch) I0917 09:19:51.461371 15634 sys_catalog.cc:340] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "abfd3dc05c4a4566bb328f04255eb266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:51.461397 15634 sys_catalog.cc:343] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [sys.catalog]: This master's current role is: FOLLOWER I0917 09:19:51.461485 15638 consensus_queue.cc:984] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s I0917 09:19:51.461849 15571 sys_catalog.cc:340] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [sys.catalog]: SysCatalogTable state changed. Reason: New leader abfd3dc05c4a4566bb328f04255eb266. Latest consensus state: current_term: 5 leader_uuid: "abfd3dc05c4a4566bb328f04255eb266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:51.461869 15559 sys_catalog.cc:340] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "abfd3dc05c4a4566bb328f04255eb266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:51.461871 15571 sys_catalog.cc:343] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [sys.catalog]: This master's current role is: FOLLOWER I0917 09:19:51.461881 15559 sys_catalog.cc:343] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [sys.catalog]: This master's current role is: FOLLOWER I0917 09:19:51.473825 15342 heartbeater.cc:322] Connected to a master server at 127.13.208.125:37503 I0917 09:19:51.474438 15270 heartbeater.cc:322] Connected to a master server at 127.13.208.125:37503 I0917 09:19:51.474510 15599 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" instance_seqno: 1568711989299382) as {username='slave'} at 127.0.0.1:51716; Asking this server to re-register. I0917 09:19:51.474675 15342 heartbeater.cc:406] Registering TS with master... I0917 09:19:51.474804 15599 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" instance_seqno: 1568711989250571) as {username='slave'} at 127.0.0.1:51712; Asking this server to re-register. I0917 09:19:51.474933 15270 heartbeater.cc:406] Registering TS with master... I0917 09:19:51.475000 15598 ts_manager.cc:161] Registered new tserver with Master: eea10e830a664ea4be74d8cffbe95552 (127.13.208.67:42065) I0917 09:19:51.475296 15598 ts_manager.cc:161] Registered new tserver with Master: f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793) I0917 09:19:51.475620 15191 heartbeater.cc:322] Connected to a master server at 127.13.208.125:37503 I0917 09:19:51.475857 15598 master_service.cc:229] Got heartbeat from unknown tserver (permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" instance_seqno: 1568711989214369) as {username='slave'} at 127.0.0.1:51714; Asking this server to re-register. I0917 09:19:51.476027 15191 heartbeater.cc:406] Registering TS with master... I0917 09:19:51.476213 15598 ts_manager.cc:161] Registered new tserver with Master: c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65:39971) I0917 09:19:51.538436 15192 heartbeater.cc:447] Master 127.13.208.126:33685 was elected leader, sending a full tablet report... I0917 09:19:51.538476 15271 heartbeater.cc:447] Master 127.13.208.126:33685 was elected leader, sending a full tablet report... I0917 09:19:51.539003 15343 heartbeater.cc:447] Master 127.13.208.126:33685 was elected leader, sending a full tablet report... I0917 09:19:51.540560 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:51.544569 14145 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0917 09:19:51.544605 14145 tablet_replica.cc:273] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a: stopping tablet replica I0917 09:19:51.544641 14145 raft_consensus.cc:2147] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 3 FOLLOWER]: Raft consensus shutting down. I0917 09:19:51.544668 14145 raft_consensus.cc:2174] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 3 FOLLOWER]: Raft consensus is shut down! I0917 09:19:51.544924 14145 tablet_replica.cc:273] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a: stopping tablet replica I0917 09:19:51.544952 14145 raft_consensus.cc:2147] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 2 LEADER]: Raft consensus shutting down. I0917 09:19:51.544996 14145 raft_consensus.cc:2174] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 2 FOLLOWER]: Raft consensus is shut down! I0917 09:19:51.545128 14145 tablet_replica.cc:273] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a: stopping tablet replica I0917 09:19:51.545151 14145 raft_consensus.cc:2147] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [term 2 LEADER]: Raft consensus shutting down. I0917 09:19:51.545189 14145 raft_consensus.cc:2174] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [term 2 FOLLOWER]: Raft consensus is shut down! I0917 09:19:51.556875 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:51.557004 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:51.557041 14145 tablet_server.cc:169] TabletServer shutdown complete. W0917 09:19:51.561324 15280 consensus_peers.cc:458] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 -> Peer c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65:39971): Couldn't send request to peer c2c8bec4d40141a6b1f470f6142de67a. Status: Network error: Client connection negotiation failed: client connection to 127.13.208.65:39971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0917 09:19:51.562381 14145 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 28304us I0917 09:19:51.562453 14145 fs_manager.cc:263] Metadata directory not provided I0917 09:19:51.562474 14145 fs_manager.cc:266] Using existing metadata directory in first data directory I0917 09:19:51.562908 14145 fs_manager.cc:399] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:51.563246 14145 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0917 09:19:51.565279 14145 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:51.565340 14145 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/ts-0-root uuid: "c2c8bec4d40141a6b1f470f6142de67a" format_stamp: "Formatted at 2019-09-17 09:19:49 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:51.565362 14145 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0917 09:19:51.567324 15638 sys_catalog.cc:340] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "abfd3dc05c4a4566bb328f04255eb266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfd3dc05c4a4566bb328f04255eb266" member_type: VOTER last_known_addr { host: "127.13.208.126" port: 33685 } } peers { permanent_uuid: "82e2633778254c44a45393ac32540f27" member_type: VOTER last_known_addr { host: "127.13.208.125" port: 37503 } } peers { permanent_uuid: "e501d8ef778246b69b34d86e7395022b" member_type: VOTER last_known_addr { host: "127.13.208.124" port: 39307 } } } I0917 09:19:51.567345 15638 sys_catalog.cc:343] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [sys.catalog]: This master's current role is: LEADER I0917 09:19:51.610143 14145 ts_tablet_manager.cc:371] Loading tablet metadata (0/3 complete) I0917 09:19:51.611121 14145 ts_tablet_manager.cc:383] Loaded tablet metadata (3 total tablets, 3 live tablets) I0917 09:19:51.611145 14145 ts_tablet_manager.cc:389] Registering tablets (0/3 complete) I0917 09:19:51.611572 15652 tablet_bootstrap.cc:439] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a: Bootstrap starting. I0917 09:19:51.612264 14145 ts_tablet_manager.cc:403] Registered 3 tablets I0917 09:19:51.613392 15652 tablet.cc:1223] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a: Alter schema from ( 0:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 0:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0917 09:19:51.613435 15652 tablet_metadata.cc:569] Not flushing: waiting for 1 pins to be released. I0917 09:19:51.613533 15652 tablet_bootstrap.cc:439] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=36 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0917 09:19:51.613775 15652 tablet_bootstrap.cc:439] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a: Bootstrap complete. I0917 09:19:51.614751 14145 rpc_server.cc:205] RPC server started. Bound to: 127.13.208.65:39971 I0917 09:19:51.615043 14145 webserver.cc:374] Webserver started at http://127.13.208.65:44321/ using document root and password file I0917 09:19:51.615221 15652 ts_tablet_manager.cc:1085] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s I0917 09:19:51.615391 15652 raft_consensus.cc:340] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:51.615417 15652 raft_consensus.cc:705] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2c8bec4d40141a6b1f470f6142de67a, State: Initialized, Role: FOLLOWER I0917 09:19:51.615507 15652 consensus_queue.cc:229] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:51.615684 15652 ts_tablet_manager.cc:1113] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s I0917 09:19:51.615725 15652 tablet_bootstrap.cc:439] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a: Bootstrap starting. I0917 09:19:51.618556 15652 tablet.cc:1223] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a: Alter schema from ( 0:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 0:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0917 09:19:51.618604 15652 tablet_metadata.cc:569] Not flushing: waiting for 1 pins to be released. I0917 09:19:51.618646 15652 tablet_bootstrap.cc:439] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=29 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0917 09:19:51.618866 15652 tablet_bootstrap.cc:439] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a: Bootstrap complete. I0917 09:19:51.620749 15710 heartbeater.cc:322] Connected to a master server at 127.13.208.125:37503 I0917 09:19:51.620791 15710 heartbeater.cc:406] Registering TS with master... I0917 09:19:51.620788 15709 heartbeater.cc:322] Connected to a master server at 127.13.208.124:39307 I0917 09:19:51.620820 15709 heartbeater.cc:406] Registering TS with master... I0917 09:19:51.620864 15710 heartbeater.cc:455] Master 127.13.208.125:37503 requested a full tablet report, sending... I0917 09:19:51.620955 15709 heartbeater.cc:455] Master 127.13.208.124:39307 requested a full tablet report, sending... I0917 09:19:51.621680 15597 ts_manager.cc:161] Re-registered known tserver with Master: c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65:39971) I0917 09:19:51.621773 15524 ts_manager.cc:161] Re-registered known tserver with Master: c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65:39971) I0917 09:19:51.622015 15652 ts_tablet_manager.cc:1085] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s I0917 09:19:51.622520 15711 heartbeater.cc:322] Connected to a master server at 127.13.208.126:33685 I0917 09:19:51.622519 15652 raft_consensus.cc:340] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 3 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } I0917 09:19:51.622550 15711 heartbeater.cc:406] Registering TS with master... I0917 09:19:51.622555 15652 raft_consensus.cc:705] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2c8bec4d40141a6b1f470f6142de67a, State: Initialized, Role: FOLLOWER I0917 09:19:51.622601 15711 heartbeater.cc:455] Master 127.13.208.126:33685 requested a full tablet report, sending... I0917 09:19:51.622648 15652 consensus_queue.cc:229] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } I0917 09:19:51.622809 15652 ts_tablet_manager.cc:1113] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0917 09:19:51.622975 15456 ts_manager.cc:161] Re-registered known tserver with Master: c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65:39971) I0917 09:19:51.623142 15652 tablet_bootstrap.cc:439] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a: Bootstrap starting. I0917 09:19:51.624354 15456 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43142 I0917 09:19:51.625145 15652 tablet.cc:1223] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a: Alter schema from ( 0:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 0:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0917 09:19:51.625169 15652 tablet_metadata.cc:569] Not flushing: waiting for 1 pins to be released. I0917 09:19:51.625185 15711 heartbeater.cc:447] Master 127.13.208.126:33685 was elected leader, sending a full tablet report... I0917 09:19:51.625191 15652 tablet_bootstrap.cc:439] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=26 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0917 09:19:51.625414 15652 tablet_bootstrap.cc:439] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a: Bootstrap complete. I0917 09:19:51.626296 15652 ts_tablet_manager.cc:1085] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s I0917 09:19:51.626425 15652 raft_consensus.cc:340] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:51.626444 15652 raft_consensus.cc:705] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2c8bec4d40141a6b1f470f6142de67a, State: Initialized, Role: FOLLOWER I0917 09:19:51.626516 15652 consensus_queue.cc:229] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:51.626715 15652 ts_tablet_manager.cc:1113] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s I0917 09:19:51.680642 15723 raft_consensus.cc:472] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2c8bec4d40141a6b1f470f6142de67a) I0917 09:19:51.680707 15723 raft_consensus.cc:494] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:51.680783 15723 leader_election.cc:296] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65:39971), eea10e830a664ea4be74d8cffbe95552 (127.13.208.67:42065) I0917 09:19:51.681001 15317 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f67125c2201a4791ab5756ecc8fba82a" candidate_uuid: "f42c56d2dc6b4faba5a218e79da90951" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "eea10e830a664ea4be74d8cffbe95552" is_pre_election: true I0917 09:19:51.681030 15317 raft_consensus.cc:2311] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f42c56d2dc6b4faba5a218e79da90951 for term 3 because replica is either leader or believes a valid leader to be alive. I0917 09:19:51.683594 15685 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f67125c2201a4791ab5756ecc8fba82a" candidate_uuid: "f42c56d2dc6b4faba5a218e79da90951" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c2c8bec4d40141a6b1f470f6142de67a" is_pre_election: true I0917 09:19:51.683638 15685 raft_consensus.cc:2350] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f42c56d2dc6b4faba5a218e79da90951 in term 2. I0917 09:19:51.683801 15200 leader_election.cc:310] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c2c8bec4d40141a6b1f470f6142de67a, f42c56d2dc6b4faba5a218e79da90951; no voters: eea10e830a664ea4be74d8cffbe95552 I0917 09:19:51.683931 15723 raft_consensus.cc:2644] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 2 FOLLOWER]: Leader pre-election won for term 3 I0917 09:19:51.683964 15723 raft_consensus.cc:472] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 2 FOLLOWER]: Starting leader election (detected failure of leader c2c8bec4d40141a6b1f470f6142de67a) I0917 09:19:51.683976 15723 raft_consensus.cc:2890] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 2 FOLLOWER]: Advancing to term 3 I0917 09:19:51.684839 15723 raft_consensus.cc:494] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:51.684976 15723 leader_election.cc:296] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [CANDIDATE]: Term 3 election: Requested vote from peers c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65:39971), eea10e830a664ea4be74d8cffbe95552 (127.13.208.67:42065) I0917 09:19:51.685191 15317 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f67125c2201a4791ab5756ecc8fba82a" candidate_uuid: "f42c56d2dc6b4faba5a218e79da90951" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "eea10e830a664ea4be74d8cffbe95552" I0917 09:19:51.685199 15685 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f67125c2201a4791ab5756ecc8fba82a" candidate_uuid: "f42c56d2dc6b4faba5a218e79da90951" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c2c8bec4d40141a6b1f470f6142de67a" I0917 09:19:51.685253 15317 raft_consensus.cc:2311] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f42c56d2dc6b4faba5a218e79da90951 for term 3 because replica is either leader or believes a valid leader to be alive. I0917 09:19:51.685278 15685 raft_consensus.cc:2890] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 2 FOLLOWER]: Advancing to term 3 I0917 09:19:51.685992 15685 raft_consensus.cc:2350] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f42c56d2dc6b4faba5a218e79da90951 in term 3. I0917 09:19:51.686143 15200 leader_election.cc:310] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c2c8bec4d40141a6b1f470f6142de67a, f42c56d2dc6b4faba5a218e79da90951; no voters: eea10e830a664ea4be74d8cffbe95552 I0917 09:19:51.686230 15723 raft_consensus.cc:2644] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 3 FOLLOWER]: Leader election won for term 3 I0917 09:19:51.686383 15723 raft_consensus.cc:667] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 3 LEADER]: Becoming Leader. State: Replica: f42c56d2dc6b4faba5a218e79da90951, State: Running, Role: LEADER I0917 09:19:51.686511 15723 consensus_queue.cc:206] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:51.687357 15456 catalog_manager.cc:4120] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 reported cstate change: term changed from 2 to 3, leader changed from c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65) to f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66). New cstate: current_term: 3 leader_uuid: "f42c56d2dc6b4faba5a218e79da90951" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } health_report { overall_health: UNKNOWN } } } I0917 09:19:51.694705 15721 raft_consensus.cc:472] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2c8bec4d40141a6b1f470f6142de67a) I0917 09:19:51.694852 15721 raft_consensus.cc:494] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:51.695083 15721 leader_election.cc:296] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793), c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65:39971) I0917 09:19:51.695204 15243 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f67125c2201a4791ab5756ecc8fba82a" candidate_uuid: "eea10e830a664ea4be74d8cffbe95552" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f42c56d2dc6b4faba5a218e79da90951" is_pre_election: true I0917 09:19:51.695264 15685 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f67125c2201a4791ab5756ecc8fba82a" candidate_uuid: "eea10e830a664ea4be74d8cffbe95552" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c2c8bec4d40141a6b1f470f6142de67a" is_pre_election: true I0917 09:19:51.695294 15243 raft_consensus.cc:2311] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate eea10e830a664ea4be74d8cffbe95552 for term 3 because replica is either leader or believes a valid leader to be alive. I0917 09:19:51.695302 15685 raft_consensus.cc:2280] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eea10e830a664ea4be74d8cffbe95552 in current term 3: Already voted for candidate f42c56d2dc6b4faba5a218e79da90951 in this term. I0917 09:19:51.695463 15279 leader_election.cc:310] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eea10e830a664ea4be74d8cffbe95552; no voters: c2c8bec4d40141a6b1f470f6142de67a, f42c56d2dc6b4faba5a218e79da90951 I0917 09:19:51.695564 15721 raft_consensus.cc:2890] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 2 FOLLOWER]: Advancing to term 3 I0917 09:19:51.696300 15721 raft_consensus.cc:2592] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0917 09:19:51.696842 15721 raft_consensus.cc:472] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2c8bec4d40141a6b1f470f6142de67a) I0917 09:19:51.697013 15721 raft_consensus.cc:494] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:51.697114 15721 leader_election.cc:296] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65:39971), f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793) I0917 09:19:51.697335 15243 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "de49f5bae5244941af0020522f04d827" candidate_uuid: "eea10e830a664ea4be74d8cffbe95552" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f42c56d2dc6b4faba5a218e79da90951" is_pre_election: true I0917 09:19:51.697371 15685 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "de49f5bae5244941af0020522f04d827" candidate_uuid: "eea10e830a664ea4be74d8cffbe95552" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c2c8bec4d40141a6b1f470f6142de67a" is_pre_election: true I0917 09:19:51.697373 15243 raft_consensus.cc:2311] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eea10e830a664ea4be74d8cffbe95552 for term 3 because replica is either leader or believes a valid leader to be alive. I0917 09:19:51.697398 15685 raft_consensus.cc:2350] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eea10e830a664ea4be74d8cffbe95552 in term 2. I0917 09:19:51.697566 15280 leader_election.cc:310] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2c8bec4d40141a6b1f470f6142de67a, eea10e830a664ea4be74d8cffbe95552; no voters: I0917 09:19:51.697724 15721 raft_consensus.cc:2644] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [term 2 FOLLOWER]: Leader pre-election won for term 3 I0917 09:19:51.697743 15721 raft_consensus.cc:472] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [term 2 FOLLOWER]: Starting leader election (detected failure of leader c2c8bec4d40141a6b1f470f6142de67a) I0917 09:19:51.697749 15721 raft_consensus.cc:2890] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [term 2 FOLLOWER]: Advancing to term 3 I0917 09:19:51.698467 15721 raft_consensus.cc:494] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:51.698535 15721 leader_election.cc:296] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [CANDIDATE]: Term 3 election: Requested vote from peers c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65:39971), f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793) I0917 09:19:51.698727 15243 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "de49f5bae5244941af0020522f04d827" candidate_uuid: "eea10e830a664ea4be74d8cffbe95552" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f42c56d2dc6b4faba5a218e79da90951" I0917 09:19:51.698738 15685 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "de49f5bae5244941af0020522f04d827" candidate_uuid: "eea10e830a664ea4be74d8cffbe95552" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c2c8bec4d40141a6b1f470f6142de67a" I0917 09:19:51.698748 15243 raft_consensus.cc:2311] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate eea10e830a664ea4be74d8cffbe95552 for term 3 because replica is either leader or believes a valid leader to be alive. I0917 09:19:51.698762 15685 raft_consensus.cc:2890] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [term 2 FOLLOWER]: Advancing to term 3 I0917 09:19:51.699363 15685 raft_consensus.cc:2350] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eea10e830a664ea4be74d8cffbe95552 in term 3. I0917 09:19:51.699561 15280 leader_election.cc:310] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c2c8bec4d40141a6b1f470f6142de67a, eea10e830a664ea4be74d8cffbe95552; no voters: f42c56d2dc6b4faba5a218e79da90951 I0917 09:19:51.699718 15721 raft_consensus.cc:2644] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [term 3 FOLLOWER]: Leader election won for term 3 I0917 09:19:51.699759 15721 raft_consensus.cc:667] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [term 3 LEADER]: Becoming Leader. State: Replica: eea10e830a664ea4be74d8cffbe95552, State: Running, Role: LEADER I0917 09:19:51.699844 15721 consensus_queue.cc:206] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:51.700649 15456 catalog_manager.cc:4120] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 reported cstate change: term changed from 2 to 3, leader changed from c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65) to eea10e830a664ea4be74d8cffbe95552 (127.13.208.67). New cstate: current_term: 3 leader_uuid: "eea10e830a664ea4be74d8cffbe95552" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } health_report { overall_health: HEALTHY } } } I0917 09:19:51.707214 15724 raft_consensus.cc:472] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2c8bec4d40141a6b1f470f6142de67a) I0917 09:19:51.707330 15724 raft_consensus.cc:494] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:51.707428 15724 leader_election.cc:296] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65:39971), eea10e830a664ea4be74d8cffbe95552 (127.13.208.67:42065) I0917 09:19:51.707654 15317 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "de49f5bae5244941af0020522f04d827" candidate_uuid: "f42c56d2dc6b4faba5a218e79da90951" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "eea10e830a664ea4be74d8cffbe95552" is_pre_election: true I0917 09:19:51.707708 15317 raft_consensus.cc:2311] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate f42c56d2dc6b4faba5a218e79da90951 for term 3 because replica is either leader or believes a valid leader to be alive. I0917 09:19:51.707832 15685 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "de49f5bae5244941af0020522f04d827" candidate_uuid: "f42c56d2dc6b4faba5a218e79da90951" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c2c8bec4d40141a6b1f470f6142de67a" is_pre_election: true I0917 09:19:51.707857 15685 raft_consensus.cc:2280] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f42c56d2dc6b4faba5a218e79da90951 in current term 3: Already voted for candidate eea10e830a664ea4be74d8cffbe95552 in this term. I0917 09:19:51.707942 15200 leader_election.cc:310] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f42c56d2dc6b4faba5a218e79da90951; no voters: c2c8bec4d40141a6b1f470f6142de67a, eea10e830a664ea4be74d8cffbe95552 I0917 09:19:51.707999 15724 raft_consensus.cc:2890] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [term 2 FOLLOWER]: Advancing to term 3 I0917 09:19:51.708748 15724 raft_consensus.cc:2592] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0917 09:19:51.766623 15685 raft_consensus.cc:1184] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 3 FOLLOWER]: Refusing update from remote peer f42c56d2dc6b4faba5a218e79da90951: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch) I0917 09:19:51.766882 15724 consensus_queue.cc:984] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0917 09:19:51.768088 15317 raft_consensus.cc:1184] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 3 FOLLOWER]: Refusing update from remote peer f42c56d2dc6b4faba5a218e79da90951: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch) I0917 09:19:51.768586 15724 consensus_queue.cc:984] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0917 09:19:51.807340 15685 raft_consensus.cc:1184] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [term 3 FOLLOWER]: Refusing update from remote peer eea10e830a664ea4be74d8cffbe95552: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch) I0917 09:19:51.807621 15411 consensus_queue.cc:984] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0917 09:19:51.809229 15243 raft_consensus.cc:1184] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [term 3 FOLLOWER]: Refusing update from remote peer eea10e830a664ea4be74d8cffbe95552: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch) I0917 09:19:51.809633 15735 consensus_queue.cc:984] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0917 09:19:51.816184 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:51.821102 14145 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0917 09:19:51.821135 14145 tablet_replica.cc:273] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: stopping tablet replica I0917 09:19:51.821164 14145 raft_consensus.cc:2147] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 3 FOLLOWER]: Raft consensus shutting down. I0917 09:19:51.821182 14145 raft_consensus.cc:2174] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 3 FOLLOWER]: Raft consensus is shut down! I0917 09:19:51.821378 14145 tablet_replica.cc:273] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951: stopping tablet replica I0917 09:19:51.821401 14145 raft_consensus.cc:2147] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 3 LEADER]: Raft consensus shutting down. I0917 09:19:51.821432 14145 raft_consensus.cc:2174] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 3 FOLLOWER]: Raft consensus is shut down! I0917 09:19:51.821662 14145 tablet_replica.cc:273] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951: stopping tablet replica I0917 09:19:51.821687 14145 raft_consensus.cc:2147] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [term 3 FOLLOWER]: Raft consensus shutting down. I0917 09:19:51.821696 14145 raft_consensus.cc:2174] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [term 3 FOLLOWER]: Raft consensus is shut down! W0917 09:19:51.831527 15279 consensus_peers.cc:458] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 -> Peer f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793): Couldn't send request to peer f42c56d2dc6b4faba5a218e79da90951. Status: Network error: Client connection negotiation failed: client connection to 127.13.208.66:38793: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0917 09:19:51.833493 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:51.833523 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:51.833542 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:51.838233 14145 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 28304us I0917 09:19:51.838304 14145 fs_manager.cc:263] Metadata directory not provided I0917 09:19:51.838320 14145 fs_manager.cc:266] Using existing metadata directory in first data directory I0917 09:19:51.838837 14145 fs_manager.cc:399] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:51.839193 14145 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0917 09:19:51.841217 14145 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:51.841281 14145 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/ts-1-root uuid: "f42c56d2dc6b4faba5a218e79da90951" format_stamp: "Formatted at 2019-09-17 09:19:49 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:51.841305 14145 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0917 09:19:51.866588 14145 ts_tablet_manager.cc:383] Loaded tablet metadata (3 total tablets, 3 live tablets) I0917 09:19:51.867465 14145 ts_tablet_manager.cc:403] Registered 3 tablets I0917 09:19:51.868963 15747 tablet_bootstrap.cc:439] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951: Bootstrap starting. I0917 09:19:51.870424 14145 rpc_server.cc:205] RPC server started. Bound to: 127.13.208.66:38793 I0917 09:19:51.870532 15747 tablet.cc:1223] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951: Alter schema from ( 0:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 0:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0917 09:19:51.870573 15747 tablet_metadata.cc:569] Not flushing: waiting for 1 pins to be released. I0917 09:19:51.870638 15747 tablet_bootstrap.cc:439] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=36 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0917 09:19:51.870745 14145 webserver.cc:374] Webserver started at http://127.13.208.66:45521/ using document root and password file I0917 09:19:51.870839 15747 tablet_bootstrap.cc:439] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951: Bootstrap complete. I0917 09:19:51.872977 15747 ts_tablet_manager.cc:1085] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s I0917 09:19:51.873246 15747 raft_consensus.cc:340] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:51.873270 15747 raft_consensus.cc:705] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: f42c56d2dc6b4faba5a218e79da90951, State: Initialized, Role: FOLLOWER I0917 09:19:51.873379 15747 consensus_queue.cc:229] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:51.873504 15747 ts_tablet_manager.cc:1113] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I0917 09:19:51.873558 15747 tablet_bootstrap.cc:439] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: Bootstrap starting. I0917 09:19:51.876001 15747 tablet.cc:1223] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: Alter schema from ( 0:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 0:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0917 09:19:51.876039 15747 tablet_metadata.cc:569] Not flushing: waiting for 1 pins to be released. I0917 09:19:51.876096 15747 tablet_bootstrap.cc:439] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=1 applied=4 ignored=0} inserts{seen=29 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0917 09:19:51.876410 15747 tablet_bootstrap.cc:439] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: Bootstrap complete. I0917 09:19:51.876719 15805 heartbeater.cc:322] Connected to a master server at 127.13.208.126:33685 I0917 09:19:51.876763 15805 heartbeater.cc:406] Registering TS with master... I0917 09:19:51.877241 15805 heartbeater.cc:455] Master 127.13.208.126:33685 requested a full tablet report, sending... I0917 09:19:51.877743 15456 ts_manager.cc:161] Re-registered known tserver with Master: f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793) I0917 09:19:51.878752 15803 heartbeater.cc:322] Connected to a master server at 127.13.208.124:39307 I0917 09:19:51.878769 15456 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43156 I0917 09:19:51.878787 15803 heartbeater.cc:406] Registering TS with master... I0917 09:19:51.878829 15803 heartbeater.cc:455] Master 127.13.208.124:39307 requested a full tablet report, sending... I0917 09:19:51.878952 15747 ts_tablet_manager.cc:1085] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s E0917 09:19:51.879150 15747 ts_tablet_manager.cc:1124] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: Tablet failed to start: Corruption: Unable to start RaftConsensus: The last op in the WAL with id 3.4 has a term (3) that is greater than the latest recorded term, which is 2 I0917 09:19:51.879180 15747 ts_tablet_manager.cc:1113] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I0917 09:19:51.879284 15747 tablet_replica.cc:273] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: stopping tablet replica I0917 09:19:51.879315 15747 raft_consensus.cc:2147] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 2 FOLLOWER]: Raft consensus shutting down. I0917 09:19:51.879343 15747 raft_consensus.cc:2174] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 2 FOLLOWER]: Raft consensus is shut down! I0917 09:19:51.879386 15523 ts_manager.cc:161] Re-registered known tserver with Master: f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793) I0917 09:19:51.879658 15805 heartbeater.cc:447] Master 127.13.208.126:33685 was elected leader, sending a full tablet report... I0917 09:19:51.879863 15804 heartbeater.cc:322] Connected to a master server at 127.13.208.125:37503 I0917 09:19:51.879922 15804 heartbeater.cc:406] Registering TS with master... I0917 09:19:51.879987 15804 heartbeater.cc:455] Master 127.13.208.125:37503 requested a full tablet report, sending... I0917 09:19:51.880131 15747 tablet_bootstrap.cc:439] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951: Bootstrap starting. W0917 09:19:51.880244 15456 catalog_manager.cc:4041] Tablet ac74b319ad54416685f8b9d9506e1d61 (table NewTableName [id=03b5e80e6ff342bab99657ab154c5e00]) has failed on TS f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793): Corruption: Unable to start RaftConsensus: The last op in the WAL with id 3.4 has a term (3) that is greater than the latest recorded term, which is 2 I0917 09:19:51.880501 15599 ts_manager.cc:161] Re-registered known tserver with Master: f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793) I0917 09:19:51.881785 15747 tablet.cc:1223] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951: Alter schema from ( 0:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 0:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0917 09:19:51.881815 15747 tablet_metadata.cc:569] Not flushing: waiting for 1 pins to be released. I0917 09:19:51.881865 15747 tablet_bootstrap.cc:439] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=26 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0917 09:19:51.882035 15747 tablet_bootstrap.cc:439] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951: Bootstrap complete. I0917 09:19:51.882851 15747 ts_tablet_manager.cc:1085] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s I0917 09:19:51.882979 15747 raft_consensus.cc:340] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:51.882998 15747 raft_consensus.cc:705] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: f42c56d2dc6b4faba5a218e79da90951, State: Initialized, Role: FOLLOWER I0917 09:19:51.883072 15747 consensus_queue.cc:229] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:51.883244 15747 ts_tablet_manager.cc:1113] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s I0917 09:19:51.938957 15279 consensus_queue.cc:544] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [LEADER]: The tablet replica hosted on peer f42c56d2dc6b4faba5a218e79da90951 has failed I0917 09:19:51.939827 15456 catalog_manager.cc:3871] Sending ChangeConfig:REMOVE_PEER on tablet ac74b319ad54416685f8b9d9506e1d61 (attempt 1) I0917 09:19:51.943123 15317 consensus_queue.cc:206] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } I0917 09:19:51.943526 15685 raft_consensus.cc:1184] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 3 FOLLOWER]: Refusing update from remote peer eea10e830a664ea4be74d8cffbe95552: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch) I0917 09:19:51.943722 15721 consensus_queue.cc:984] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0917 09:19:51.944278 15721 raft_consensus.cc:2792] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index -1 to 5, VOTER f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } } I0917 09:19:51.944394 15685 raft_consensus.cc:2792] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index -1 to 5, VOTER f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } } I0917 09:19:51.946494 15440 catalog_manager.cc:3654] ChangeConfig:REMOVE_PEER RPC for tablet ac74b319ad54416685f8b9d9506e1d61 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0917 09:19:51.947013 15455 catalog_manager.cc:4120] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a reported cstate change: config changed from index -1 to 5, VOTER f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66) evicted. New cstate: current_term: 3 leader_uuid: "eea10e830a664ea4be74d8cffbe95552" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } } I0917 09:19:51.948097 15455 catalog_manager.cc:3512] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet ac74b319ad54416685f8b9d9506e1d61 on f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793) (TS f42c56d2dc6b4faba5a218e79da90951 not found in new config with opid_index 5) I0917 09:19:51.948416 15456 catalog_manager.cc:3739] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet ac74b319ad54416685f8b9d9506e1d61 (attempt 1) I0917 09:19:51.948750 15317 consensus_queue.cc:206] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: NON_VOTER last_known_addr { host: "127.13.208.66" port: 38793 } attrs { promote: true } } I0917 09:19:51.949136 15685 raft_consensus.cc:1184] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 3 FOLLOWER]: Refusing update from remote peer eea10e830a664ea4be74d8cffbe95552: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch) I0917 09:19:51.949165 15279 consensus_queue.cc:544] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [LEADER]: The tablet replica hosted on peer f42c56d2dc6b4faba5a218e79da90951 has failed W0917 09:19:51.949204 15279 consensus_peers.cc:458] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 -> Peer f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793): Couldn't send request to peer f42c56d2dc6b4faba5a218e79da90951. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: Corruption: Unable to start RaftConsensus: The last op in the WAL with id 3.4 has a term (3) that is greater than the latest recorded term, which is 2. This is attempt 1: this message will repeat every 5th retry. I0917 09:19:51.949317 15411 consensus_queue.cc:984] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I0917 09:19:51.950341 15735 raft_consensus.cc:2792] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, NON_VOTER f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: NON_VOTER last_known_addr { host: "127.13.208.66" port: 38793 } attrs { promote: true } } } I0917 09:19:51.950450 15685 raft_consensus.cc:2792] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, NON_VOTER f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: NON_VOTER last_known_addr { host: "127.13.208.66" port: 38793 } attrs { promote: true } } } I0917 09:19:51.952543 15440 catalog_manager.cc:3654] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ac74b319ad54416685f8b9d9506e1d61 with cas_config_opid_index 5: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0917 09:19:51.953564 15456 catalog_manager.cc:4120] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 reported cstate change: config changed from index 5 to 6, NON_VOTER f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66) added. New cstate: current_term: 3 leader_uuid: "eea10e830a664ea4be74d8cffbe95552" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: NON_VOTER last_known_addr { host: "127.13.208.66" port: 38793 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } } I0917 09:19:51.954466 15769 tablet_service.cc:1066] Processing DeleteTablet for tablet ac74b319ad54416685f8b9d9506e1d61 with delete_type TABLET_DATA_TOMBSTONED (TS f42c56d2dc6b4faba5a218e79da90951 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:45276 I0917 09:19:51.954766 15456 catalog_manager.cc:3871] Sending ChangeConfig:REMOVE_PEER on tablet ac74b319ad54416685f8b9d9506e1d61 (attempt 1) I0917 09:19:51.954849 15819 ts_tablet_manager.cc:1440] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0917 09:19:51.955404 15317 consensus_queue.cc:206] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } I0917 09:19:51.955747 15685 raft_consensus.cc:1184] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 3 FOLLOWER]: Refusing update from remote peer eea10e830a664ea4be74d8cffbe95552: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch) I0917 09:19:51.955996 15721 consensus_queue.cc:984] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0917 09:19:51.956640 15735 raft_consensus.cc:2792] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [term 3 LEADER]: Committing config change with OpId 3.7: config changed from index 6 to 7, NON_VOTER f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } } I0917 09:19:51.956925 15685 raft_consensus.cc:2792] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 6 to 7, NON_VOTER f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } } I0917 09:19:51.957206 15819 ts_tablet_manager.cc:1453] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4 I0917 09:19:51.957273 15819 log.cc:990] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: Deleting WAL directory at /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/minicluster-data/ts-1-root/wals/ac74b319ad54416685f8b9d9506e1d61 I0917 09:19:51.957789 15437 catalog_manager.cc:3490] TS f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793): tablet ac74b319ad54416685f8b9d9506e1d61 (table NewTableName [id=03b5e80e6ff342bab99657ab154c5e00]) successfully deleted I0917 09:19:51.958559 15440 catalog_manager.cc:3654] ChangeConfig:REMOVE_PEER RPC for tablet ac74b319ad54416685f8b9d9506e1d61 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0917 09:19:51.959345 15456 catalog_manager.cc:4120] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a reported cstate change: config changed from index 6 to 7, NON_VOTER f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66) evicted. New cstate: current_term: 3 leader_uuid: "eea10e830a664ea4be74d8cffbe95552" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } } I0917 09:19:51.960542 15456 catalog_manager.cc:3512] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet ac74b319ad54416685f8b9d9506e1d61 on f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793) (TS f42c56d2dc6b4faba5a218e79da90951 not found in new config with opid_index 7) I0917 09:19:51.960690 15455 catalog_manager.cc:3739] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet ac74b319ad54416685f8b9d9506e1d61 (attempt 1) I0917 09:19:51.961028 15769 tablet_service.cc:1066] Processing DeleteTablet for tablet ac74b319ad54416685f8b9d9506e1d61 with delete_type TABLET_DATA_TOMBSTONED (TS f42c56d2dc6b4faba5a218e79da90951 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:45276 I0917 09:19:51.961067 15317 consensus_queue.cc:206] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: NON_VOTER last_known_addr { host: "127.13.208.66" port: 38793 } attrs { promote: true } } I0917 09:19:51.961169 15437 catalog_manager.cc:3490] TS f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793): tablet ac74b319ad54416685f8b9d9506e1d61 (table NewTableName [id=03b5e80e6ff342bab99657ab154c5e00]) successfully deleted I0917 09:19:51.961405 15685 raft_consensus.cc:1184] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 3 FOLLOWER]: Refusing update from remote peer eea10e830a664ea4be74d8cffbe95552: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch) W0917 09:19:51.961462 15279 consensus_peers.cc:458] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 -> Peer f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793): Couldn't send request to peer f42c56d2dc6b4faba5a218e79da90951. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry. I0917 09:19:51.961607 15735 consensus_queue.cc:984] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I0917 09:19:51.962143 15411 raft_consensus.cc:2792] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [term 3 LEADER]: Committing config change with OpId 3.8: config changed from index 7 to 8, NON_VOTER f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: NON_VOTER last_known_addr { host: "127.13.208.66" port: 38793 } attrs { promote: true } } } I0917 09:19:51.962265 15685 raft_consensus.cc:2792] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 7 to 8, NON_VOTER f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: NON_VOTER last_known_addr { host: "127.13.208.66" port: 38793 } attrs { promote: true } } } I0917 09:19:51.963994 15440 catalog_manager.cc:3654] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ac74b319ad54416685f8b9d9506e1d61 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0917 09:19:51.964570 15455 catalog_manager.cc:4120] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 reported cstate change: config changed from index 7 to 8, NON_VOTER f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66) added. New cstate: current_term: 3 leader_uuid: "eea10e830a664ea4be74d8cffbe95552" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: NON_VOTER last_known_addr { host: "127.13.208.66" port: 38793 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0917 09:19:51.968160 15714 raft_consensus.cc:472] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f42c56d2dc6b4faba5a218e79da90951) I0917 09:19:51.968237 15714 raft_consensus.cc:494] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:51.968361 15714 leader_election.cc:296] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793), eea10e830a664ea4be74d8cffbe95552 (127.13.208.67:42065) I0917 09:19:51.969609 15735 raft_consensus.cc:472] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f42c56d2dc6b4faba5a218e79da90951) I0917 09:19:51.969715 15735 raft_consensus.cc:494] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:51.969830 15735 leader_election.cc:296] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793), c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65:39971) I0917 09:19:51.970093 15779 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f67125c2201a4791ab5756ecc8fba82a" candidate_uuid: "eea10e830a664ea4be74d8cffbe95552" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f42c56d2dc6b4faba5a218e79da90951" is_pre_election: true I0917 09:19:51.970155 15779 raft_consensus.cc:2350] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eea10e830a664ea4be74d8cffbe95552 in term 3. I0917 09:19:51.970252 15279 leader_election.cc:310] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eea10e830a664ea4be74d8cffbe95552, f42c56d2dc6b4faba5a218e79da90951; no voters: I0917 09:19:51.970295 15735 raft_consensus.cc:2644] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 3 FOLLOWER]: Leader pre-election won for term 4 I0917 09:19:51.970307 15735 raft_consensus.cc:472] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f42c56d2dc6b4faba5a218e79da90951) I0917 09:19:51.970324 15685 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f67125c2201a4791ab5756ecc8fba82a" candidate_uuid: "eea10e830a664ea4be74d8cffbe95552" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c2c8bec4d40141a6b1f470f6142de67a" is_pre_election: true I0917 09:19:51.970326 15735 raft_consensus.cc:2890] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 3 FOLLOWER]: Advancing to term 4 I0917 09:19:51.970374 15685 raft_consensus.cc:2350] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eea10e830a664ea4be74d8cffbe95552 in term 3. I0917 09:19:51.971046 15735 raft_consensus.cc:494] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:51.971316 15779 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f67125c2201a4791ab5756ecc8fba82a" candidate_uuid: "eea10e830a664ea4be74d8cffbe95552" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f42c56d2dc6b4faba5a218e79da90951" I0917 09:19:51.971350 15779 raft_consensus.cc:2890] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 3 FOLLOWER]: Advancing to term 4 I0917 09:19:51.971356 15735 leader_election.cc:296] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [CANDIDATE]: Term 4 election: Requested vote from peers f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66:38793), c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65:39971) I0917 09:19:51.971453 15685 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f67125c2201a4791ab5756ecc8fba82a" candidate_uuid: "eea10e830a664ea4be74d8cffbe95552" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c2c8bec4d40141a6b1f470f6142de67a" I0917 09:19:51.971472 15685 raft_consensus.cc:2890] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 3 FOLLOWER]: Advancing to term 4 I0917 09:19:51.973188 15685 raft_consensus.cc:2350] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eea10e830a664ea4be74d8cffbe95552 in term 4. I0917 09:19:51.973350 15280 leader_election.cc:310] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2c8bec4d40141a6b1f470f6142de67a, eea10e830a664ea4be74d8cffbe95552; no voters: I0917 09:19:51.973543 15735 raft_consensus.cc:2644] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 4 FOLLOWER]: Leader election won for term 4 I0917 09:19:51.973608 15735 raft_consensus.cc:667] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 4 LEADER]: Becoming Leader. State: Replica: eea10e830a664ea4be74d8cffbe95552, State: Running, Role: LEADER I0917 09:19:51.973750 15735 consensus_queue.cc:206] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } I0917 09:19:51.974210 15779 raft_consensus.cc:2350] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eea10e830a664ea4be74d8cffbe95552 in term 4. I0917 09:19:51.974678 15779 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f67125c2201a4791ab5756ecc8fba82a" candidate_uuid: "c2c8bec4d40141a6b1f470f6142de67a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f42c56d2dc6b4faba5a218e79da90951" is_pre_election: true I0917 09:19:51.974736 15779 raft_consensus.cc:2280] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c2c8bec4d40141a6b1f470f6142de67a in current term 4: Already voted for candidate eea10e830a664ea4be74d8cffbe95552 in this term. I0917 09:19:51.974942 15456 catalog_manager.cc:4120] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 reported cstate change: term changed from 3 to 4, leader changed from f42c56d2dc6b4faba5a218e79da90951 (127.13.208.66) to eea10e830a664ea4be74d8cffbe95552 (127.13.208.67). New cstate: current_term: 4 leader_uuid: "eea10e830a664ea4be74d8cffbe95552" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } health_report { overall_health: HEALTHY } } } I0917 09:19:51.975551 15317 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f67125c2201a4791ab5756ecc8fba82a" candidate_uuid: "c2c8bec4d40141a6b1f470f6142de67a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eea10e830a664ea4be74d8cffbe95552" is_pre_election: true I0917 09:19:51.975584 15317 raft_consensus.cc:2311] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate c2c8bec4d40141a6b1f470f6142de67a for term 4 because replica is either leader or believes a valid leader to be alive. I0917 09:19:51.975714 15647 leader_election.cc:310] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2c8bec4d40141a6b1f470f6142de67a; no voters: eea10e830a664ea4be74d8cffbe95552, f42c56d2dc6b4faba5a218e79da90951 I0917 09:19:51.975824 15714 raft_consensus.cc:2592] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority I0917 09:19:52.049535 15685 raft_consensus.cc:1184] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 4 FOLLOWER]: Refusing update from remote peer eea10e830a664ea4be74d8cffbe95552: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch) I0917 09:19:52.049832 15735 consensus_queue.cc:984] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I0917 09:19:52.051134 15779 raft_consensus.cc:1184] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 4 FOLLOWER]: Refusing update from remote peer eea10e830a664ea4be74d8cffbe95552: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch) I0917 09:19:52.051635 15721 consensus_queue.cc:984] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 38793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s /home/jenkins-slave/workspace/kudu-master/1/src/kudu/integration-tests/ts_tablet_manager-itest.cc:753: Failure Failed Bad status: Corruption: Unable to start RaftConsensus: The last op in the WAL with id 3.4 has a term (3) that is greater than the latest recorded term, which is 2 I0917 09:19:52.071379 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:52.074326 15821 ts_tablet_manager.cc:708] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: Initiating tablet copy from peer eea10e830a664ea4be74d8cffbe95552 (127.13.208.67:42065) I0917 09:19:52.074424 15821 tablet_copy_client.cc:241] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: tablet copy: Beginning tablet copy session from remote peer at address 127.13.208.67:42065 I0917 09:19:52.075729 14145 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0917 09:19:52.075886 14145 tablet_replica.cc:273] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a: stopping tablet replica I0917 09:19:52.075933 14145 raft_consensus.cc:2147] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [term 3 FOLLOWER]: Raft consensus shutting down. I0917 09:19:52.075959 14145 raft_consensus.cc:2174] T de49f5bae5244941af0020522f04d827 P c2c8bec4d40141a6b1f470f6142de67a [term 3 FOLLOWER]: Raft consensus is shut down! I0917 09:19:52.076333 14145 tablet_replica.cc:273] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a: stopping tablet replica I0917 09:19:52.076371 14145 raft_consensus.cc:2147] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 3 FOLLOWER]: Raft consensus shutting down. I0917 09:19:52.076386 14145 raft_consensus.cc:2174] T ac74b319ad54416685f8b9d9506e1d61 P c2c8bec4d40141a6b1f470f6142de67a [term 3 FOLLOWER]: Raft consensus is shut down! I0917 09:19:52.076709 14145 tablet_replica.cc:273] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a: stopping tablet replica I0917 09:19:52.076746 14145 raft_consensus.cc:2147] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 4 FOLLOWER]: Raft consensus shutting down. I0917 09:19:52.076759 14145 raft_consensus.cc:2174] T f67125c2201a4791ab5756ecc8fba82a P c2c8bec4d40141a6b1f470f6142de67a [term 4 FOLLOWER]: Raft consensus is shut down! I0917 09:19:52.078888 15328 tablet_copy_service.cc:135] P eea10e830a664ea4be74d8cffbe95552: Received BeginTabletCopySession request for tablet ac74b319ad54416685f8b9d9506e1d61 from peer f42c56d2dc6b4faba5a218e79da90951 ({username='slave'} at 127.0.0.1:58882) I0917 09:19:52.078922 15328 tablet_copy_service.cc:156] P eea10e830a664ea4be74d8cffbe95552: Beginning new tablet copy session on tablet ac74b319ad54416685f8b9d9506e1d61 from peer f42c56d2dc6b4faba5a218e79da90951 at {username='slave'} at 127.0.0.1:58882: session id = f42c56d2dc6b4faba5a218e79da90951-ac74b319ad54416685f8b9d9506e1d61 I0917 09:19:52.079592 15328 tablet_copy_source_session.cc:202] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552: Tablet Copy: opened 0 blocks and 1 log segments I0917 09:19:52.079908 15821 ts_tablet_manager.cc:1440] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: Deleting tablet data with delete state TABLET_DATA_COPYING I0917 09:19:52.081665 15821 ts_tablet_manager.cc:1453] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 3.4 I0917 09:19:52.081737 15821 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet ac74b319ad54416685f8b9d9506e1d61. 1 dirs total, 0 dirs full, 0 dirs failed I0917 09:19:52.082319 15821 tablet_copy_client.cc:579] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: tablet copy: Starting download of 0 data blocks... I0917 09:19:52.082480 15821 tablet_copy_client.cc:542] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: tablet copy: Starting download of 1 WAL segments... I0917 09:19:52.082882 15821 tablet_copy_client.cc:418] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0917 09:19:52.083447 15821 tablet_bootstrap.cc:439] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: Bootstrap starting. W0917 09:19:52.084744 15280 consensus_peers.cc:458] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 -> Peer c2c8bec4d40141a6b1f470f6142de67a (127.13.208.65:39971): Couldn't send request to peer c2c8bec4d40141a6b1f470f6142de67a. Status: Network error: Client connection negotiation failed: client connection to 127.13.208.65:39971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0917 09:19:52.084861 15821 tablet.cc:1223] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: Alter schema from ( 0:key INT32 NOT NULL, PRIMARY KEY (key) ) version 0 to ( 0:key INT32 NOT NULL, PRIMARY KEY (key) ) version 1 I0917 09:19:52.084926 15821 tablet_metadata.cc:569] Not flushing: waiting for 1 pins to be released. I0917 09:19:52.085093 15821 tablet_bootstrap.cc:439] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=29 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0917 09:19:52.085234 15821 tablet_bootstrap.cc:439] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: Bootstrap complete. I0917 09:19:52.085882 15821 ts_tablet_manager.cc:1085] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s I0917 09:19:52.086050 15821 raft_consensus.cc:340] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 3 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: NON_VOTER last_known_addr { host: "127.13.208.66" port: 38793 } attrs { promote: true } } I0917 09:19:52.086069 15821 raft_consensus.cc:705] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: f42c56d2dc6b4faba5a218e79da90951, State: Initialized, Role: LEARNER I0917 09:19:52.086109 15821 consensus_queue.cc:229] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eea10e830a664ea4be74d8cffbe95552" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 42065 } } peers { permanent_uuid: "c2c8bec4d40141a6b1f470f6142de67a" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39971 } } peers { permanent_uuid: "f42c56d2dc6b4faba5a218e79da90951" member_type: NON_VOTER last_known_addr { host: "127.13.208.66" port: 38793 } attrs { promote: true } } I0917 09:19:52.086293 15821 ts_tablet_manager.cc:1113] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s I0917 09:19:52.086525 15328 tablet_copy_service.cc:325] P eea10e830a664ea4be74d8cffbe95552: Request end of tablet copy session f42c56d2dc6b4faba5a218e79da90951-ac74b319ad54416685f8b9d9506e1d61 received from {username='slave'} at 127.0.0.1:58882 I0917 09:19:52.086555 15328 tablet_copy_service.cc:413] P eea10e830a664ea4be74d8cffbe95552: ending tablet copy session f42c56d2dc6b4faba5a218e79da90951-ac74b319ad54416685f8b9d9506e1d61 on tablet ac74b319ad54416685f8b9d9506e1d61 with peer f42c56d2dc6b4faba5a218e79da90951 I0917 09:19:52.089099 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:52.089120 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:52.089143 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:52.089993 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:52.092597 14145 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0917 09:19:52.092722 14145 tablet_replica.cc:273] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951: stopping tablet replica I0917 09:19:52.092763 14145 raft_consensus.cc:2147] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [term 3 FOLLOWER]: Raft consensus shutting down. I0917 09:19:52.092813 14145 raft_consensus.cc:2174] T de49f5bae5244941af0020522f04d827 P f42c56d2dc6b4faba5a218e79da90951 [term 3 FOLLOWER]: Raft consensus is shut down! I0917 09:19:52.093211 14145 tablet_replica.cc:273] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951: stopping tablet replica I0917 09:19:52.093245 14145 raft_consensus.cc:2147] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 3 LEARNER]: Raft consensus shutting down. I0917 09:19:52.093261 14145 raft_consensus.cc:2174] T ac74b319ad54416685f8b9d9506e1d61 P f42c56d2dc6b4faba5a218e79da90951 [term 3 LEARNER]: Raft consensus is shut down! I0917 09:19:52.093484 14145 tablet_replica.cc:273] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951: stopping tablet replica I0917 09:19:52.093508 14145 raft_consensus.cc:2147] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 4 FOLLOWER]: Raft consensus shutting down. I0917 09:19:52.093528 14145 raft_consensus.cc:2174] T f67125c2201a4791ab5756ecc8fba82a P f42c56d2dc6b4faba5a218e79da90951 [term 4 FOLLOWER]: Raft consensus is shut down! I0917 09:19:52.105399 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:52.105434 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:52.105458 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:52.106525 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:52.109647 14145 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0917 09:19:52.109678 14145 tablet_replica.cc:273] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552: stopping tablet replica I0917 09:19:52.109710 14145 raft_consensus.cc:2147] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [term 3 LEADER]: Raft consensus shutting down. I0917 09:19:52.109761 14145 raft_consensus.cc:2174] T ac74b319ad54416685f8b9d9506e1d61 P eea10e830a664ea4be74d8cffbe95552 [term 3 FOLLOWER]: Raft consensus is shut down! I0917 09:19:52.110066 14145 tablet_replica.cc:273] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552: stopping tablet replica I0917 09:19:52.110096 14145 raft_consensus.cc:2147] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 4 LEADER]: Raft consensus shutting down. I0917 09:19:52.110139 14145 raft_consensus.cc:2174] T f67125c2201a4791ab5756ecc8fba82a P eea10e830a664ea4be74d8cffbe95552 [term 4 FOLLOWER]: Raft consensus is shut down! I0917 09:19:52.110378 14145 tablet_replica.cc:273] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552: stopping tablet replica I0917 09:19:52.110399 14145 raft_consensus.cc:2147] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [term 3 LEADER]: Raft consensus shutting down. I0917 09:19:52.110426 14145 raft_consensus.cc:2174] T de49f5bae5244941af0020522f04d827 P eea10e830a664ea4be74d8cffbe95552 [term 3 FOLLOWER]: Raft consensus is shut down! W0917 09:19:52.112064 14145 rolling_log.cc:179] Unable to compress old log file: Not found: Unable to open input file to compress: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145/ts_tablet_manager-itest.dist-test-slave-dist-test-slave-w6nz.slave.diagnostics.20190917-091949.5.14145: No such file or directory (error 2) I0917 09:19:52.112222 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:52.112241 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:52.112260 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:52.113144 14145 master.cc:247] Master@127.13.208.126:33685 shutting down... I0917 09:19:52.115298 14145 raft_consensus.cc:2147] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 5 LEADER]: Raft consensus shutting down. I0917 09:19:52.115378 14145 raft_consensus.cc:2174] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266 [term 5 FOLLOWER]: Raft consensus is shut down! I0917 09:19:52.115393 14145 tablet_replica.cc:273] T 00000000000000000000000000000000 P abfd3dc05c4a4566bb328f04255eb266: stopping tablet replica I0917 09:19:52.127208 14145 master.cc:258] Master@127.13.208.126:33685 shutdown complete. I0917 09:19:52.128077 14145 master.cc:247] Master@127.13.208.125:37503 shutting down... I0917 09:19:52.130472 14145 raft_consensus.cc:2147] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 5 FOLLOWER]: Raft consensus shutting down. I0917 09:19:52.130506 14145 raft_consensus.cc:2174] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27 [term 5 FOLLOWER]: Raft consensus is shut down! I0917 09:19:52.130517 14145 tablet_replica.cc:273] T 00000000000000000000000000000000 P 82e2633778254c44a45393ac32540f27: stopping tablet replica I0917 09:19:52.142038 14145 master.cc:258] Master@127.13.208.125:37503 shutdown complete. I0917 09:19:52.142829 14145 master.cc:247] Master@127.13.208.124:39307 shutting down... I0917 09:19:52.145438 14145 raft_consensus.cc:2147] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 5 FOLLOWER]: Raft consensus shutting down. I0917 09:19:52.145475 14145 raft_consensus.cc:2174] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b [term 5 FOLLOWER]: Raft consensus is shut down! I0917 09:19:52.145488 14145 tablet_replica.cc:273] T 00000000000000000000000000000000 P e501d8ef778246b69b34d86e7395022b: stopping tablet replica I0917 09:19:52.157256 14145 master.cc:258] Master@127.13.208.124:39307 shutdown complete. I0917 09:19:52.158440 14145 test_util.cc:135] ----------------------------------------------- I0917 09:19:52.158469 14145 test_util.cc:136] Had fatal failures, leaving test files at /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1568711978778636-14145 [ FAILED ] TsTabletManagerITest.TestTableStats (3155 ms) [----------] 4 tests from TsTabletManagerITest (13375 ms total) [----------] 2 tests from FailedTabletsAreReplacedITest [ RUN ] FailedTabletsAreReplacedITest.OneReplica/0 I0917 09:19:52.159493 14145 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.13.208.126:45123 I0917 09:19:52.160279 14145 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 28804us I0917 09:19:52.160329 14145 fs_manager.cc:263] Metadata directory not provided I0917 09:19:52.160356 14145 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0917 09:19:52.160383 14145 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0917 09:19:52.160394 14145 server_base.cc:453] Attempting to create new FS layout instead I0917 09:19:52.160634 14145 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568711978778636-14145/minicluster-data/master-0-root/instance: uuid: "ebc3e15792a94db9b35e13b731cdd6f0" format_stamp: "Formatted at 2019-09-17 09:19:52 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:52.161682 14145 fs_manager.cc:503] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s I0917 09:19:52.161963 14145 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0917 09:19:52.162551 14145 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:52.162595 14145 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568711978778636-14145/minicluster-data/master-0-root uuid: "ebc3e15792a94db9b35e13b731cdd6f0" format_stamp: "Formatted at 2019-09-17 09:19:52 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:52.162611 14145 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568711978778636-14145/minicluster-data/master-0-root metadata directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568711978778636-14145/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568711978778636-14145/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0917 09:19:52.180392 14145 rpc_server.cc:205] RPC server started. Bound to: 127.13.208.126:45123 I0917 09:19:52.180577 14145 webserver.cc:374] Webserver started at http://127.13.208.126:42991/ using document root and password file I0917 09:19:52.180944 15882 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0917 09:19:52.181944 15882 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P ebc3e15792a94db9b35e13b731cdd6f0: Bootstrap starting. I0917 09:19:52.182186 15882 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P ebc3e15792a94db9b35e13b731cdd6f0: No blocks or log segments found. Creating new log. I0917 09:19:52.182634 15882 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P ebc3e15792a94db9b35e13b731cdd6f0: No bootstrap required, opened a new log I0917 09:19:52.182785 15882 raft_consensus.cc:340] T 00000000000000000000000000000000 P ebc3e15792a94db9b35e13b731cdd6f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc3e15792a94db9b35e13b731cdd6f0" member_type: VOTER } I0917 09:19:52.182801 15882 raft_consensus.cc:366] T 00000000000000000000000000000000 P ebc3e15792a94db9b35e13b731cdd6f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0917 09:19:52.182806 15882 raft_consensus.cc:705] T 00000000000000000000000000000000 P ebc3e15792a94db9b35e13b731cdd6f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebc3e15792a94db9b35e13b731cdd6f0, State: Initialized, Role: FOLLOWER I0917 09:19:52.182849 15882 consensus_queue.cc:229] T 00000000000000000000000000000000 P ebc3e15792a94db9b35e13b731cdd6f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc3e15792a94db9b35e13b731cdd6f0" member_type: VOTER } I0917 09:19:52.182857 15882 raft_consensus.cc:380] T 00000000000000000000000000000000 P ebc3e15792a94db9b35e13b731cdd6f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0917 09:19:52.182875 15882 raft_consensus.cc:472] T 00000000000000000000000000000000 P ebc3e15792a94db9b35e13b731cdd6f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0917 09:19:52.182884 15882 raft_consensus.cc:2890] T 00000000000000000000000000000000 P ebc3e15792a94db9b35e13b731cdd6f0 [term 0 FOLLOWER]: Advancing to term 1 I0917 09:19:52.183365 15882 raft_consensus.cc:494] T 00000000000000000000000000000000 P ebc3e15792a94db9b35e13b731cdd6f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc3e15792a94db9b35e13b731cdd6f0" member_type: VOTER } I0917 09:19:52.183401 15882 leader_election.cc:310] T 00000000000000000000000000000000 P ebc3e15792a94db9b35e13b731cdd6f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebc3e15792a94db9b35e13b731cdd6f0; no voters: I0917 09:19:52.183473 15882 leader_election.cc:296] T 00000000000000000000000000000000 P ebc3e15792a94db9b35e13b731cdd6f0 [CANDIDATE]: Term 1 election: Requested vote from peers I0917 09:19:52.183527 15884 raft_consensus.cc:2644] T 00000000000000000000000000000000 P ebc3e15792a94db9b35e13b731cdd6f0 [term 1 FOLLOWER]: Leader election won for term 1 I0917 09:19:52.183614 15884 raft_consensus.cc:667] T 00000000000000000000000000000000 P ebc3e15792a94db9b35e13b731cdd6f0 [term 1 LEADER]: Becoming Leader. State: Replica: ebc3e15792a94db9b35e13b731cdd6f0, State: Running, Role: LEADER I0917 09:19:52.183670 15884 consensus_queue.cc:206] T 00000000000000000000000000000000 P ebc3e15792a94db9b35e13b731cdd6f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc3e15792a94db9b35e13b731cdd6f0" member_type: VOTER } I0917 09:19:52.183806 15884 sys_catalog.cc:340] T 00000000000000000000000000000000 P ebc3e15792a94db9b35e13b731cdd6f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebc3e15792a94db9b35e13b731cdd6f0. Latest consensus state: current_term: 1 leader_uuid: "ebc3e15792a94db9b35e13b731cdd6f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc3e15792a94db9b35e13b731cdd6f0" member_type: VOTER } } I0917 09:19:52.183828 15884 sys_catalog.cc:343] T 00000000000000000000000000000000 P ebc3e15792a94db9b35e13b731cdd6f0 [sys.catalog]: This master's current role is: LEADER I0917 09:19:52.183833 15882 sys_catalog.cc:340] T 00000000000000000000000000000000 P ebc3e15792a94db9b35e13b731cdd6f0 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "ebc3e15792a94db9b35e13b731cdd6f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc3e15792a94db9b35e13b731cdd6f0" member_type: VOTER } } I0917 09:19:52.183847 15882 sys_catalog.cc:343] T 00000000000000000000000000000000 P ebc3e15792a94db9b35e13b731cdd6f0 [sys.catalog]: This master's current role is: LEADER I0917 09:19:52.183918 15885 sys_catalog.cc:340] T 00000000000000000000000000000000 P ebc3e15792a94db9b35e13b731cdd6f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebc3e15792a94db9b35e13b731cdd6f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc3e15792a94db9b35e13b731cdd6f0" member_type: VOTER } } I0917 09:19:52.183940 15885 sys_catalog.cc:343] T 00000000000000000000000000000000 P ebc3e15792a94db9b35e13b731cdd6f0 [sys.catalog]: This master's current role is: LEADER I0917 09:19:52.183974 15882 sys_catalog.cc:428] T 00000000000000000000000000000000 P ebc3e15792a94db9b35e13b731cdd6f0 [sys.catalog]: configured and running, proceeding with master startup. I0917 09:19:52.184084 14145 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 W0917 09:19:52.184451 15889 catalog_manager.cc:1144] T 00000000000000000000000000000000 P ebc3e15792a94db9b35e13b731cdd6f0: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0917 09:19:52.184507 15889 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0917 09:19:52.184535 15888 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0917 09:19:52.184639 15888 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0917 09:19:52.197136 15888 catalog_manager.cc:981] Generated new certificate authority record I0917 09:19:52.197670 15888 catalog_manager.cc:1106] Loading token signing keys... I0917 09:19:52.205868 15888 catalog_manager.cc:4435] T 00000000000000000000000000000000 P ebc3e15792a94db9b35e13b731cdd6f0: Generated new TSK 0 I0917 09:19:52.216943 14145 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 28804us I0917 09:19:52.217016 14145 fs_manager.cc:263] Metadata directory not provided I0917 09:19:52.217042 14145 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0917 09:19:52.217080 14145 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0917 09:19:52.217090 14145 server_base.cc:453] Attempting to create new FS layout instead I0917 09:19:52.217334 14145 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568711978778636-14145/minicluster-data/ts-0-root/instance: uuid: "613d003c41ff4c86a70ce0634c46d578" format_stamp: "Formatted at 2019-09-17 09:19:52 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:52.218513 14145 fs_manager.cc:503] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s I0917 09:19:52.218783 14145 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0917 09:19:52.219419 14145 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:52.219455 14145 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568711978778636-14145/minicluster-data/ts-0-root uuid: "613d003c41ff4c86a70ce0634c46d578" format_stamp: "Formatted at 2019-09-17 09:19:52 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:52.219476 14145 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568711978778636-14145/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568711978778636-14145/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568711978778636-14145/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0917 09:19:52.249233 14145 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0917 09:19:52.249259 14145 ts_tablet_manager.cc:403] Registered 0 tablets I0917 09:19:52.251395 14145 rpc_server.cc:205] RPC server started. Bound to: 127.13.208.65:33935 I0917 09:19:52.251591 14145 webserver.cc:374] Webserver started at http://127.13.208.65:39443/ using document root and password file I0917 09:19:52.253441 14145 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 28804us I0917 09:19:52.253509 14145 fs_manager.cc:263] Metadata directory not provided I0917 09:19:52.253540 14145 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0917 09:19:52.253577 14145 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0917 09:19:52.253589 14145 server_base.cc:453] Attempting to create new FS layout instead I0917 09:19:52.253824 14145 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568711978778636-14145/minicluster-data/ts-1-root/instance: uuid: "bf5537cd74844bf389b64a96222d16f0" format_stamp: "Formatted at 2019-09-17 09:19:52 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:52.254869 15956 heartbeater.cc:322] Connected to a master server at 127.13.208.126:45123 I0917 09:19:52.254897 15956 heartbeater.cc:406] Registering TS with master... I0917 09:19:52.254943 15956 heartbeater.cc:455] Master 127.13.208.126:45123 requested a full tablet report, sending... I0917 09:19:52.255121 15849 ts_manager.cc:161] Registered new tserver with Master: 613d003c41ff4c86a70ce0634c46d578 (127.13.208.65:33935) I0917 09:19:52.255376 14145 fs_manager.cc:503] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s I0917 09:19:52.255604 14145 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0917 09:19:52.255910 15849 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49788 I0917 09:19:52.256469 14145 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:52.256507 14145 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568711978778636-14145/minicluster-data/ts-1-root uuid: "bf5537cd74844bf389b64a96222d16f0" format_stamp: "Formatted at 2019-09-17 09:19:52 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:52.256531 14145 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568711978778636-14145/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568711978778636-14145/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568711978778636-14145/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0917 09:19:52.291549 14145 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0917 09:19:52.291580 14145 ts_tablet_manager.cc:403] Registered 0 tablets I0917 09:19:52.293592 14145 rpc_server.cc:205] RPC server started. Bound to: 127.13.208.66:40967 I0917 09:19:52.293825 14145 webserver.cc:374] Webserver started at http://127.13.208.66:37229/ using document root and password file I0917 09:19:52.296470 14145 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 28804us I0917 09:19:52.296547 14145 fs_manager.cc:263] Metadata directory not provided I0917 09:19:52.296579 14145 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0917 09:19:52.296617 14145 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0917 09:19:52.296624 14145 server_base.cc:453] Attempting to create new FS layout instead I0917 09:19:52.297132 14145 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568711978778636-14145/minicluster-data/ts-2-root/instance: uuid: "f3112f5bdce74833a3bdec2091deb1a5" format_stamp: "Formatted at 2019-09-17 09:19:52 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:52.297590 16024 heartbeater.cc:322] Connected to a master server at 127.13.208.126:45123 I0917 09:19:52.297619 16024 heartbeater.cc:406] Registering TS with master... I0917 09:19:52.297672 16024 heartbeater.cc:455] Master 127.13.208.126:45123 requested a full tablet report, sending... I0917 09:19:52.297842 15849 ts_manager.cc:161] Registered new tserver with Master: bf5537cd74844bf389b64a96222d16f0 (127.13.208.66:40967) I0917 09:19:52.298277 15849 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49790 I0917 09:19:52.298847 14145 fs_manager.cc:503] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s I0917 09:19:52.299111 14145 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0917 09:19:52.300320 14145 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:52.300371 14145 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568711978778636-14145/minicluster-data/ts-2-root uuid: "f3112f5bdce74833a3bdec2091deb1a5" format_stamp: "Formatted at 2019-09-17 09:19:52 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:52.300388 14145 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568711978778636-14145/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568711978778636-14145/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568711978778636-14145/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0917 09:19:52.312292 14145 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0917 09:19:52.312319 14145 ts_tablet_manager.cc:403] Registered 0 tablets I0917 09:19:52.314594 14145 rpc_server.cc:205] RPC server started. Bound to: 127.13.208.67:36533 I0917 09:19:52.314843 14145 webserver.cc:374] Webserver started at http://127.13.208.67:37417/ using document root and password file I0917 09:19:52.317814 16091 heartbeater.cc:322] Connected to a master server at 127.13.208.126:45123 I0917 09:19:52.317848 16091 heartbeater.cc:406] Registering TS with master... I0917 09:19:52.317931 16091 heartbeater.cc:455] Master 127.13.208.126:45123 requested a full tablet report, sending... I0917 09:19:52.318114 15849 ts_manager.cc:161] Registered new tserver with Master: f3112f5bdce74833a3bdec2091deb1a5 (127.13.208.67:36533) I0917 09:19:52.318245 14145 internal_mini_cluster.cc:328] 3 TS(s) registered with all masters after 0.00329701s I0917 09:19:52.318550 15849 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49792 I0917 09:19:52.322216 14145 test_util.cc:212] Using random seed: 957261990 I0917 09:19:52.326696 15849 catalog_manager.cc:1440] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49796: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0917 09:19:52.333093 15986 tablet_service.cc:1022] Processing CreateTablet for tablet 93a32378223c423392af40eb79c3e2b9 (table=test-workload [id=6ce6072f86684f7ab7fc2b10ba4b0948]), partition=RANGE (key) PARTITION UNBOUNDED I0917 09:19:52.333266 15986 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 93a32378223c423392af40eb79c3e2b9. 1 dirs total, 0 dirs full, 0 dirs failed I0917 09:19:52.333456 15923 tablet_service.cc:1022] Processing CreateTablet for tablet 93a32378223c423392af40eb79c3e2b9 (table=test-workload [id=6ce6072f86684f7ab7fc2b10ba4b0948]), partition=RANGE (key) PARTITION UNBOUNDED I0917 09:19:52.333528 15923 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 93a32378223c423392af40eb79c3e2b9. 1 dirs total, 0 dirs full, 0 dirs failed I0917 09:19:52.334681 16058 tablet_service.cc:1022] Processing CreateTablet for tablet 93a32378223c423392af40eb79c3e2b9 (table=test-workload [id=6ce6072f86684f7ab7fc2b10ba4b0948]), partition=RANGE (key) PARTITION UNBOUNDED I0917 09:19:52.334822 16058 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 93a32378223c423392af40eb79c3e2b9. 1 dirs total, 0 dirs full, 0 dirs failed I0917 09:19:52.336946 16111 tablet_bootstrap.cc:439] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578: Bootstrap starting. I0917 09:19:52.337018 16112 tablet_bootstrap.cc:439] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0: Bootstrap starting. I0917 09:19:52.337604 16111 tablet_bootstrap.cc:590] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578: No blocks or log segments found. Creating new log. I0917 09:19:52.337646 16112 tablet_bootstrap.cc:590] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0: No blocks or log segments found. Creating new log. I0917 09:19:52.337739 16113 tablet_bootstrap.cc:439] T 93a32378223c423392af40eb79c3e2b9 P f3112f5bdce74833a3bdec2091deb1a5: Bootstrap starting. I0917 09:19:52.337983 16113 tablet_bootstrap.cc:590] T 93a32378223c423392af40eb79c3e2b9 P f3112f5bdce74833a3bdec2091deb1a5: No blocks or log segments found. Creating new log. I0917 09:19:52.338392 16112 tablet_bootstrap.cc:439] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0: No bootstrap required, opened a new log I0917 09:19:52.338461 16112 ts_tablet_manager.cc:1085] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I0917 09:19:52.338532 16113 tablet_bootstrap.cc:439] T 93a32378223c423392af40eb79c3e2b9 P f3112f5bdce74833a3bdec2091deb1a5: No bootstrap required, opened a new log I0917 09:19:52.338546 16111 tablet_bootstrap.cc:439] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578: No bootstrap required, opened a new log I0917 09:19:52.338569 16111 ts_tablet_manager.cc:1085] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I0917 09:19:52.338614 16113 ts_tablet_manager.cc:1085] T 93a32378223c423392af40eb79c3e2b9 P f3112f5bdce74833a3bdec2091deb1a5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I0917 09:19:52.338873 16113 raft_consensus.cc:340] T 93a32378223c423392af40eb79c3e2b9 P f3112f5bdce74833a3bdec2091deb1a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3112f5bdce74833a3bdec2091deb1a5" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 36533 } } peers { permanent_uuid: "613d003c41ff4c86a70ce0634c46d578" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 33935 } } peers { permanent_uuid: "bf5537cd74844bf389b64a96222d16f0" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40967 } } I0917 09:19:52.338909 16113 raft_consensus.cc:366] T 93a32378223c423392af40eb79c3e2b9 P f3112f5bdce74833a3bdec2091deb1a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0917 09:19:52.338928 16113 raft_consensus.cc:705] T 93a32378223c423392af40eb79c3e2b9 P f3112f5bdce74833a3bdec2091deb1a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3112f5bdce74833a3bdec2091deb1a5, State: Initialized, Role: FOLLOWER I0917 09:19:52.338980 16113 consensus_queue.cc:229] T 93a32378223c423392af40eb79c3e2b9 P f3112f5bdce74833a3bdec2091deb1a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3112f5bdce74833a3bdec2091deb1a5" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 36533 } } peers { permanent_uuid: "613d003c41ff4c86a70ce0634c46d578" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 33935 } } peers { permanent_uuid: "bf5537cd74844bf389b64a96222d16f0" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40967 } } I0917 09:19:52.339005 16112 raft_consensus.cc:340] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3112f5bdce74833a3bdec2091deb1a5" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 36533 } } peers { permanent_uuid: "613d003c41ff4c86a70ce0634c46d578" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 33935 } } peers { permanent_uuid: "bf5537cd74844bf389b64a96222d16f0" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40967 } } I0917 09:19:52.339027 16112 raft_consensus.cc:366] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0917 09:19:52.339035 16112 raft_consensus.cc:705] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf5537cd74844bf389b64a96222d16f0, State: Initialized, Role: FOLLOWER I0917 09:19:52.339007 16111 raft_consensus.cc:340] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3112f5bdce74833a3bdec2091deb1a5" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 36533 } } peers { permanent_uuid: "613d003c41ff4c86a70ce0634c46d578" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 33935 } } peers { permanent_uuid: "bf5537cd74844bf389b64a96222d16f0" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40967 } } I0917 09:19:52.339078 16111 raft_consensus.cc:366] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0917 09:19:52.339093 16111 raft_consensus.cc:705] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 613d003c41ff4c86a70ce0634c46d578, State: Initialized, Role: FOLLOWER I0917 09:19:52.339107 16091 heartbeater.cc:447] Master 127.13.208.126:45123 was elected leader, sending a full tablet report... I0917 09:19:52.339229 16112 consensus_queue.cc:229] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3112f5bdce74833a3bdec2091deb1a5" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 36533 } } peers { permanent_uuid: "613d003c41ff4c86a70ce0634c46d578" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 33935 } } peers { permanent_uuid: "bf5537cd74844bf389b64a96222d16f0" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40967 } } I0917 09:19:52.339283 16113 ts_tablet_manager.cc:1113] T 93a32378223c423392af40eb79c3e2b9 P f3112f5bdce74833a3bdec2091deb1a5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0917 09:19:52.339377 16024 heartbeater.cc:447] Master 127.13.208.126:45123 was elected leader, sending a full tablet report... I0917 09:19:52.339380 16112 ts_tablet_manager.cc:1113] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0917 09:19:52.339489 16111 consensus_queue.cc:229] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3112f5bdce74833a3bdec2091deb1a5" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 36533 } } peers { permanent_uuid: "613d003c41ff4c86a70ce0634c46d578" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 33935 } } peers { permanent_uuid: "bf5537cd74844bf389b64a96222d16f0" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40967 } } I0917 09:19:52.339666 15956 heartbeater.cc:447] Master 127.13.208.126:45123 was elected leader, sending a full tablet report... I0917 09:19:52.339874 16111 ts_tablet_manager.cc:1113] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0917 09:19:52.706907 16118 raft_consensus.cc:472] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0917 09:19:52.707021 16118 raft_consensus.cc:494] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3112f5bdce74833a3bdec2091deb1a5" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 36533 } } peers { permanent_uuid: "613d003c41ff4c86a70ce0634c46d578" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 33935 } } peers { permanent_uuid: "bf5537cd74844bf389b64a96222d16f0" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40967 } } I0917 09:19:52.707123 16118 leader_election.cc:296] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3112f5bdce74833a3bdec2091deb1a5 (127.13.208.67:36533), 613d003c41ff4c86a70ce0634c46d578 (127.13.208.65:33935) I0917 09:19:52.712211 15933 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "93a32378223c423392af40eb79c3e2b9" candidate_uuid: "bf5537cd74844bf389b64a96222d16f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "613d003c41ff4c86a70ce0634c46d578" is_pre_election: true I0917 09:19:52.712216 16068 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "93a32378223c423392af40eb79c3e2b9" candidate_uuid: "bf5537cd74844bf389b64a96222d16f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3112f5bdce74833a3bdec2091deb1a5" is_pre_election: true I0917 09:19:52.712323 15933 raft_consensus.cc:2350] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf5537cd74844bf389b64a96222d16f0 in term 0. I0917 09:19:52.712323 16068 raft_consensus.cc:2350] T 93a32378223c423392af40eb79c3e2b9 P f3112f5bdce74833a3bdec2091deb1a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf5537cd74844bf389b64a96222d16f0 in term 0. I0917 09:19:52.712453 15965 leader_election.cc:310] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 613d003c41ff4c86a70ce0634c46d578, bf5537cd74844bf389b64a96222d16f0; no voters: I0917 09:19:52.712532 16118 raft_consensus.cc:2644] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0917 09:19:52.712558 16118 raft_consensus.cc:472] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0917 09:19:52.712571 16118 raft_consensus.cc:2890] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0 [term 0 FOLLOWER]: Advancing to term 1 I0917 09:19:52.713322 16118 raft_consensus.cc:494] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3112f5bdce74833a3bdec2091deb1a5" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 36533 } } peers { permanent_uuid: "613d003c41ff4c86a70ce0634c46d578" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 33935 } } peers { permanent_uuid: "bf5537cd74844bf389b64a96222d16f0" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40967 } } I0917 09:19:52.713405 16118 leader_election.cc:296] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0 [CANDIDATE]: Term 1 election: Requested vote from peers f3112f5bdce74833a3bdec2091deb1a5 (127.13.208.67:36533), 613d003c41ff4c86a70ce0634c46d578 (127.13.208.65:33935) I0917 09:19:52.713570 15933 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "93a32378223c423392af40eb79c3e2b9" candidate_uuid: "bf5537cd74844bf389b64a96222d16f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "613d003c41ff4c86a70ce0634c46d578" I0917 09:19:52.713573 16068 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "93a32378223c423392af40eb79c3e2b9" candidate_uuid: "bf5537cd74844bf389b64a96222d16f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3112f5bdce74833a3bdec2091deb1a5" I0917 09:19:52.713600 15933 raft_consensus.cc:2890] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 [term 0 FOLLOWER]: Advancing to term 1 I0917 09:19:52.713611 16068 raft_consensus.cc:2890] T 93a32378223c423392af40eb79c3e2b9 P f3112f5bdce74833a3bdec2091deb1a5 [term 0 FOLLOWER]: Advancing to term 1 I0917 09:19:52.714732 16068 raft_consensus.cc:2350] T 93a32378223c423392af40eb79c3e2b9 P f3112f5bdce74833a3bdec2091deb1a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf5537cd74844bf389b64a96222d16f0 in term 1. I0917 09:19:52.714735 15933 raft_consensus.cc:2350] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf5537cd74844bf389b64a96222d16f0 in term 1. I0917 09:19:52.714881 15962 leader_election.cc:310] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf5537cd74844bf389b64a96222d16f0, f3112f5bdce74833a3bdec2091deb1a5; no voters: I0917 09:19:52.714953 16118 raft_consensus.cc:2644] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0 [term 1 FOLLOWER]: Leader election won for term 1 I0917 09:19:52.715023 16118 raft_consensus.cc:667] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0 [term 1 LEADER]: Becoming Leader. State: Replica: bf5537cd74844bf389b64a96222d16f0, State: Running, Role: LEADER I0917 09:19:52.715102 16118 consensus_queue.cc:206] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3112f5bdce74833a3bdec2091deb1a5" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 36533 } } peers { permanent_uuid: "613d003c41ff4c86a70ce0634c46d578" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 33935 } } peers { permanent_uuid: "bf5537cd74844bf389b64a96222d16f0" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40967 } } I0917 09:19:52.715595 15848 catalog_manager.cc:4120] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0 reported cstate change: term changed from 0 to 1, leader changed from to bf5537cd74844bf389b64a96222d16f0 (127.13.208.66). New cstate: current_term: 1 leader_uuid: "bf5537cd74844bf389b64a96222d16f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3112f5bdce74833a3bdec2091deb1a5" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 36533 } } peers { permanent_uuid: "613d003c41ff4c86a70ce0634c46d578" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 33935 } } peers { permanent_uuid: "bf5537cd74844bf389b64a96222d16f0" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40967 } } } Master Summary UUID | Address | Status ----------------------------------+----------------------+--------- ebc3e15792a94db9b35e13b731cdd6f0 | 127.13.208.126:45123 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+---------------------+---------+---------- 613d003c41ff4c86a70ce0634c46d578 | 127.13.208.65:33935 | HEALTHY | bf5537cd74844bf389b64a96222d16f0 | 127.13.208.66:40967 | HEALTHY | f3112f5bdce74833a3bdec2091deb1a5 | 127.13.208.67:36533 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 4 server(s) checked Tablet Summary Tablet 93a32378223c423392af40eb79c3e2b9 of table 'test-workload' is conflicted: 5 replicas' active configs disagree with the leader master's f3112f5bdce74833a3bdec2091deb1a5 (127.13.208.67:36533): RUNNING 613d003c41ff4c86a70ce0634c46d578 (127.13.208.65:33935): RUNNING bf5537cd74844bf389b64a96222d16f0 (127.13.208.66:40967): RUNNING [LEADER] All reported replicas are: A = f3112f5bdce74833a3bdec2091deb1a5 B = 613d003c41ff4c86a70ce0634c46d578 C = bf5537cd74844bf389b64a96222d16f0 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C | 1 | -1 | Yes B | A B C | 1 | -1 | Yes C | A B C* | 1 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0917 09:19:52.795240 14145 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors Master Summary UUID | Address | Status ----------------------------------+----------------------+--------- ebc3e15792a94db9b35e13b731cdd6f0 | 127.13.208.126:45123 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+---------------------+---------+---------- 613d003c41ff4c86a70ce0634c46d578 | 127.13.208.65:33935 | HEALTHY | bf5537cd74844bf389b64a96222d16f0 | 127.13.208.66:40967 | HEALTHY | f3112f5bdce74833a3bdec2091deb1a5 | 127.13.208.67:36533 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 4 server(s) checked Tablet Summary Tablet 93a32378223c423392af40eb79c3e2b9 of table 'test-workload' is conflicted: 5 replicas' active configs disagree with the leader master's f3112f5bdce74833a3bdec2091deb1a5 (127.13.208.67:36533): RUNNING 613d003c41ff4c86a70ce0634c46d578 (127.13.208.65:33935): RUNNING bf5537cd74844bf389b64a96222d16f0 (127.13.208.66:40967): RUNNING [LEADER] All reported replicas are: A = f3112f5bdce74833a3bdec2091deb1a5 B = 613d003c41ff4c86a70ce0634c46d578 C = bf5537cd74844bf389b64a96222d16f0 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C | 1 | -1 | Yes B | A B C | 1 | -1 | Yes C | A B C* | 1 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0917 09:19:52.971864 14145 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0917 09:19:53.096385 16118 consensus_queue.cc:984] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "613d003c41ff4c86a70ce0634c46d578" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0917 09:19:53.097682 16118 consensus_queue.cc:984] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3112f5bdce74833a3bdec2091deb1a5" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 36533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s Master Summary UUID | Address | Status ----------------------------------+----------------------+--------- ebc3e15792a94db9b35e13b731cdd6f0 | 127.13.208.126:45123 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+---------------------+---------+---------- 613d003c41ff4c86a70ce0634c46d578 | 127.13.208.65:33935 | HEALTHY | bf5537cd74844bf389b64a96222d16f0 | 127.13.208.66:40967 | HEALTHY | f3112f5bdce74833a3bdec2091deb1a5 | 127.13.208.67:36533 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 4 server(s) checked Tablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+---------+---------------+---------+------------+------------------+------------- test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0917 09:19:53.223716 14145 log_verifier.cc:121] Checking tablet 93a32378223c423392af40eb79c3e2b9 I0917 09:19:53.245980 14145 log_verifier.cc:172] Verified matching terms for 1 ops in tablet 93a32378223c423392af40eb79c3e2b9 I0917 09:19:53.246093 14145 tablet_replica.cc:273] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0: stopping tablet replica I0917 09:19:53.246141 14145 raft_consensus.cc:2147] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0 [term 1 LEADER]: Raft consensus shutting down. I0917 09:19:53.246213 14145 raft_consensus.cc:2174] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0 [term 1 FOLLOWER]: Raft consensus is shut down! Master Summary UUID | Address | Status ----------------------------------+----------------------+--------- ebc3e15792a94db9b35e13b731cdd6f0 | 127.13.208.126:45123 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+---------------------+---------+---------- 613d003c41ff4c86a70ce0634c46d578 | 127.13.208.65:33935 | HEALTHY | bf5537cd74844bf389b64a96222d16f0 | 127.13.208.66:40967 | HEALTHY | f3112f5bdce74833a3bdec2091deb1a5 | 127.13.208.67:36533 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 4 server(s) checked Tablet Summary Tablet 93a32378223c423392af40eb79c3e2b9 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING f3112f5bdce74833a3bdec2091deb1a5 (127.13.208.67:36533): RUNNING 613d003c41ff4c86a70ce0634c46d578 (127.13.208.65:33935): RUNNING bf5537cd74844bf389b64a96222d16f0 (127.13.208.66:40967): not running [LEADER] State: FAILED Data state: TABLET_DATA_READY Last status: IO error: INJECTED ERROR: tablet failed All reported replicas are: A = f3112f5bdce74833a3bdec2091deb1a5 B = 613d003c41ff4c86a70ce0634c46d578 C = bf5537cd74844bf389b64a96222d16f0 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C* | 1 | -1 | Yes B | A B C* | 1 | -1 | Yes C | [config not available] | | | Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0917 09:19:53.273667 14145 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors Master Summary UUID | Address | Status ----------------------------------+----------------------+--------- ebc3e15792a94db9b35e13b731cdd6f0 | 127.13.208.126:45123 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+---------------------+---------+---------- 613d003c41ff4c86a70ce0634c46d578 | 127.13.208.65:33935 | HEALTHY | bf5537cd74844bf389b64a96222d16f0 | 127.13.208.66:40967 | HEALTHY | f3112f5bdce74833a3bdec2091deb1a5 | 127.13.208.67:36533 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 4 server(s) checked Tablet Summary Tablet 93a32378223c423392af40eb79c3e2b9 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING f3112f5bdce74833a3bdec2091deb1a5 (127.13.208.67:36533): RUNNING 613d003c41ff4c86a70ce0634c46d578 (127.13.208.65:33935): RUNNING bf5537cd74844bf389b64a96222d16f0 (127.13.208.66:40967): not running [LEADER] State: FAILED Data state: TABLET_DATA_READY Last status: IO error: INJECTED ERROR: tablet failed All reported replicas are: A = f3112f5bdce74833a3bdec2091deb1a5 B = 613d003c41ff4c86a70ce0634c46d578 C = bf5537cd74844bf389b64a96222d16f0 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C* | 1 | -1 | Yes B | A B C* | 1 | -1 | Yes C | [config not available] | | | Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0917 09:19:53.449901 14145 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors Master Summary UUID | Address | Status ----------------------------------+----------------------+--------- ebc3e15792a94db9b35e13b731cdd6f0 | 127.13.208.126:45123 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+---------------------+---------+---------- 613d003c41ff4c86a70ce0634c46d578 | 127.13.208.65:33935 | HEALTHY | bf5537cd74844bf389b64a96222d16f0 | 127.13.208.66:40967 | HEALTHY | f3112f5bdce74833a3bdec2091deb1a5 | 127.13.208.67:36533 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 4 server(s) checked Tablet Summary Tablet 93a32378223c423392af40eb79c3e2b9 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING f3112f5bdce74833a3bdec2091deb1a5 (127.13.208.67:36533): RUNNING 613d003c41ff4c86a70ce0634c46d578 (127.13.208.65:33935): RUNNING bf5537cd74844bf389b64a96222d16f0 (127.13.208.66:40967): not running [LEADER] State: FAILED Data state: TABLET_DATA_READY Last status: IO error: INJECTED ERROR: tablet failed All reported replicas are: A = f3112f5bdce74833a3bdec2091deb1a5 B = 613d003c41ff4c86a70ce0634c46d578 C = bf5537cd74844bf389b64a96222d16f0 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C* | 1 | -1 | Yes B | A B C* | 1 | -1 | Yes C | [config not available] | | | Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0917 09:19:53.700098 14145 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors Master Summary UUID | Address | Status ----------------------------------+----------------------+--------- ebc3e15792a94db9b35e13b731cdd6f0 | 127.13.208.126:45123 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+---------------------+---------+---------- 613d003c41ff4c86a70ce0634c46d578 | 127.13.208.65:33935 | HEALTHY | bf5537cd74844bf389b64a96222d16f0 | 127.13.208.66:40967 | HEALTHY | f3112f5bdce74833a3bdec2091deb1a5 | 127.13.208.67:36533 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 4 server(s) checked Tablet Summary Tablet 93a32378223c423392af40eb79c3e2b9 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING f3112f5bdce74833a3bdec2091deb1a5 (127.13.208.67:36533): RUNNING 613d003c41ff4c86a70ce0634c46d578 (127.13.208.65:33935): RUNNING bf5537cd74844bf389b64a96222d16f0 (127.13.208.66:40967): not running [LEADER] State: FAILED Data state: TABLET_DATA_READY Last status: IO error: INJECTED ERROR: tablet failed All reported replicas are: A = f3112f5bdce74833a3bdec2091deb1a5 B = 613d003c41ff4c86a70ce0634c46d578 C = bf5537cd74844bf389b64a96222d16f0 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C* | 1 | -1 | Yes B | A B C* | 1 | -1 | Yes C | [config not available] | | | Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0917 09:19:54.063403 14145 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors Master Summary UUID | Address | Status ----------------------------------+----------------------+--------- ebc3e15792a94db9b35e13b731cdd6f0 | 127.13.208.126:45123 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+---------------------+---------+---------- 613d003c41ff4c86a70ce0634c46d578 | 127.13.208.65:33935 | HEALTHY | bf5537cd74844bf389b64a96222d16f0 | 127.13.208.66:40967 | HEALTHY | f3112f5bdce74833a3bdec2091deb1a5 | 127.13.208.67:36533 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 4 server(s) checked Tablet Summary Tablet 93a32378223c423392af40eb79c3e2b9 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING f3112f5bdce74833a3bdec2091deb1a5 (127.13.208.67:36533): RUNNING 613d003c41ff4c86a70ce0634c46d578 (127.13.208.65:33935): RUNNING bf5537cd74844bf389b64a96222d16f0 (127.13.208.66:40967): not running [LEADER] State: FAILED Data state: TABLET_DATA_READY Last status: IO error: INJECTED ERROR: tablet failed All reported replicas are: A = f3112f5bdce74833a3bdec2091deb1a5 B = 613d003c41ff4c86a70ce0634c46d578 C = bf5537cd74844bf389b64a96222d16f0 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C* | 1 | -1 | Yes B | A B C* | 1 | -1 | Yes C | [config not available] | | | Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0917 09:19:54.596112 14145 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0917 09:19:54.598271 16306 raft_consensus.cc:472] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bf5537cd74844bf389b64a96222d16f0) I0917 09:19:54.598407 16306 raft_consensus.cc:494] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3112f5bdce74833a3bdec2091deb1a5" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 36533 } } peers { permanent_uuid: "613d003c41ff4c86a70ce0634c46d578" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 33935 } } peers { permanent_uuid: "bf5537cd74844bf389b64a96222d16f0" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40967 } } I0917 09:19:54.598506 16306 leader_election.cc:296] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f3112f5bdce74833a3bdec2091deb1a5 (127.13.208.67:36533), bf5537cd74844bf389b64a96222d16f0 (127.13.208.66:40967) I0917 09:19:54.598781 16309 raft_consensus.cc:472] T 93a32378223c423392af40eb79c3e2b9 P f3112f5bdce74833a3bdec2091deb1a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bf5537cd74844bf389b64a96222d16f0) I0917 09:19:54.598858 16309 raft_consensus.cc:494] T 93a32378223c423392af40eb79c3e2b9 P f3112f5bdce74833a3bdec2091deb1a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3112f5bdce74833a3bdec2091deb1a5" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 36533 } } peers { permanent_uuid: "613d003c41ff4c86a70ce0634c46d578" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 33935 } } peers { permanent_uuid: "bf5537cd74844bf389b64a96222d16f0" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40967 } } I0917 09:19:54.599375 16309 leader_election.cc:296] T 93a32378223c423392af40eb79c3e2b9 P f3112f5bdce74833a3bdec2091deb1a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 613d003c41ff4c86a70ce0634c46d578 (127.13.208.65:33935), bf5537cd74844bf389b64a96222d16f0 (127.13.208.66:40967) I0917 09:19:54.604125 16068 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "93a32378223c423392af40eb79c3e2b9" candidate_uuid: "613d003c41ff4c86a70ce0634c46d578" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f3112f5bdce74833a3bdec2091deb1a5" is_pre_election: true I0917 09:19:54.604185 16068 raft_consensus.cc:2350] T 93a32378223c423392af40eb79c3e2b9 P f3112f5bdce74833a3bdec2091deb1a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 613d003c41ff4c86a70ce0634c46d578 in term 1. I0917 09:19:54.604336 15894 leader_election.cc:310] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 613d003c41ff4c86a70ce0634c46d578, f3112f5bdce74833a3bdec2091deb1a5; no voters: I0917 09:19:54.604609 16306 raft_consensus.cc:2644] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0917 09:19:54.604751 16306 raft_consensus.cc:472] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bf5537cd74844bf389b64a96222d16f0) I0917 09:19:54.604821 16306 raft_consensus.cc:2890] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 [term 1 FOLLOWER]: Advancing to term 2 I0917 09:19:54.604956 16001 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "93a32378223c423392af40eb79c3e2b9" candidate_uuid: "613d003c41ff4c86a70ce0634c46d578" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bf5537cd74844bf389b64a96222d16f0" is_pre_election: true I0917 09:19:54.604986 15933 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "93a32378223c423392af40eb79c3e2b9" candidate_uuid: "f3112f5bdce74833a3bdec2091deb1a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "613d003c41ff4c86a70ce0634c46d578" is_pre_election: true W0917 09:19:54.605178 15895 leader_election.cc:348] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bf5537cd74844bf389b64a96222d16f0 (127.13.208.66:40967): Illegal state: cannot vote while shut down I0917 09:19:54.605818 16306 raft_consensus.cc:494] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3112f5bdce74833a3bdec2091deb1a5" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 36533 } } peers { permanent_uuid: "613d003c41ff4c86a70ce0634c46d578" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 33935 } } peers { permanent_uuid: "bf5537cd74844bf389b64a96222d16f0" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40967 } } I0917 09:19:54.605887 15933 raft_consensus.cc:2280] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f3112f5bdce74833a3bdec2091deb1a5 in current term 2: Already voted for candidate 613d003c41ff4c86a70ce0634c46d578 in this term. I0917 09:19:54.606006 16306 leader_election.cc:296] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 [CANDIDATE]: Term 2 election: Requested vote from peers f3112f5bdce74833a3bdec2091deb1a5 (127.13.208.67:36533), bf5537cd74844bf389b64a96222d16f0 (127.13.208.66:40967) I0917 09:19:54.606086 16068 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "93a32378223c423392af40eb79c3e2b9" candidate_uuid: "613d003c41ff4c86a70ce0634c46d578" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f3112f5bdce74833a3bdec2091deb1a5" I0917 09:19:54.606110 16001 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "93a32378223c423392af40eb79c3e2b9" candidate_uuid: "613d003c41ff4c86a70ce0634c46d578" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bf5537cd74844bf389b64a96222d16f0" I0917 09:19:54.606135 16068 raft_consensus.cc:2890] T 93a32378223c423392af40eb79c3e2b9 P f3112f5bdce74833a3bdec2091deb1a5 [term 1 FOLLOWER]: Advancing to term 2 W0917 09:19:54.606338 15895 leader_election.cc:348] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer bf5537cd74844bf389b64a96222d16f0 (127.13.208.66:40967): Illegal state: cannot vote while shut down I0917 09:19:54.606791 16068 raft_consensus.cc:2350] T 93a32378223c423392af40eb79c3e2b9 P f3112f5bdce74833a3bdec2091deb1a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 613d003c41ff4c86a70ce0634c46d578 in term 2. I0917 09:19:54.606850 16001 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "93a32378223c423392af40eb79c3e2b9" candidate_uuid: "f3112f5bdce74833a3bdec2091deb1a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bf5537cd74844bf389b64a96222d16f0" is_pre_election: true I0917 09:19:54.606954 15894 leader_election.cc:310] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 613d003c41ff4c86a70ce0634c46d578, f3112f5bdce74833a3bdec2091deb1a5; no voters: bf5537cd74844bf389b64a96222d16f0 W0917 09:19:54.606971 16030 leader_election.cc:348] T 93a32378223c423392af40eb79c3e2b9 P f3112f5bdce74833a3bdec2091deb1a5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bf5537cd74844bf389b64a96222d16f0 (127.13.208.66:40967): Illegal state: cannot vote while shut down I0917 09:19:54.606992 16030 leader_election.cc:310] T 93a32378223c423392af40eb79c3e2b9 P f3112f5bdce74833a3bdec2091deb1a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f3112f5bdce74833a3bdec2091deb1a5; no voters: 613d003c41ff4c86a70ce0634c46d578, bf5537cd74844bf389b64a96222d16f0 I0917 09:19:54.607036 16306 raft_consensus.cc:2644] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 [term 2 FOLLOWER]: Leader election won for term 2 I0917 09:19:54.607053 16309 raft_consensus.cc:2592] T 93a32378223c423392af40eb79c3e2b9 P f3112f5bdce74833a3bdec2091deb1a5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0917 09:19:54.607163 16306 raft_consensus.cc:667] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 [term 2 LEADER]: Becoming Leader. State: Replica: 613d003c41ff4c86a70ce0634c46d578, State: Running, Role: LEADER I0917 09:19:54.607225 16306 consensus_queue.cc:206] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3112f5bdce74833a3bdec2091deb1a5" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 36533 } } peers { permanent_uuid: "613d003c41ff4c86a70ce0634c46d578" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 33935 } } peers { permanent_uuid: "bf5537cd74844bf389b64a96222d16f0" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40967 } } I0917 09:19:54.607807 15848 catalog_manager.cc:4120] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 reported cstate change: term changed from 1 to 2, leader changed from bf5537cd74844bf389b64a96222d16f0 (127.13.208.66) to 613d003c41ff4c86a70ce0634c46d578 (127.13.208.65). New cstate: current_term: 2 leader_uuid: "613d003c41ff4c86a70ce0634c46d578" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3112f5bdce74833a3bdec2091deb1a5" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 36533 } } peers { permanent_uuid: "613d003c41ff4c86a70ce0634c46d578" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 33935 } } peers { permanent_uuid: "bf5537cd74844bf389b64a96222d16f0" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40967 } } } W0917 09:19:55.092998 15895 consensus_peers.cc:458] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 -> Peer bf5537cd74844bf389b64a96222d16f0 (127.13.208.66:40967): Couldn't send request to peer bf5537cd74844bf389b64a96222d16f0. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry. I0917 09:19:55.099896 16068 raft_consensus.cc:1184] T 93a32378223c423392af40eb79c3e2b9 P f3112f5bdce74833a3bdec2091deb1a5 [term 2 FOLLOWER]: Refusing update from remote peer 613d003c41ff4c86a70ce0634c46d578: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch) I0917 09:19:55.100186 16313 consensus_queue.cc:984] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3112f5bdce74833a3bdec2091deb1a5" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 36533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s Master Summary UUID | Address | Status ----------------------------------+----------------------+--------- ebc3e15792a94db9b35e13b731cdd6f0 | 127.13.208.126:45123 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+---------------------+---------+---------- 613d003c41ff4c86a70ce0634c46d578 | 127.13.208.65:33935 | HEALTHY | bf5537cd74844bf389b64a96222d16f0 | 127.13.208.66:40967 | HEALTHY | f3112f5bdce74833a3bdec2091deb1a5 | 127.13.208.67:36533 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 4 server(s) checked Tablet Summary Tablet 93a32378223c423392af40eb79c3e2b9 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING f3112f5bdce74833a3bdec2091deb1a5 (127.13.208.67:36533): RUNNING 613d003c41ff4c86a70ce0634c46d578 (127.13.208.65:33935): RUNNING [LEADER] bf5537cd74844bf389b64a96222d16f0 (127.13.208.66:40967): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: INJECTED ERROR: tablet failed All reported replicas are: A = f3112f5bdce74833a3bdec2091deb1a5 B = 613d003c41ff4c86a70ce0634c46d578 C = bf5537cd74844bf389b64a96222d16f0 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 2 | -1 | Yes B | A B* C | 2 | -1 | Yes C | [config not available] | | | Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0917 09:19:55.382778 14145 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0917 09:19:55.492189 16313 raft_consensus.cc:922] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578: Attempting to remove follower bf5537cd74844bf389b64a96222d16f0 from the Raft config. Reason: The tablet replica hosted on peer bf5537cd74844bf389b64a96222d16f0 has failed I0917 09:19:55.492422 16313 consensus_queue.cc:206] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f3112f5bdce74833a3bdec2091deb1a5" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 36533 } } peers { permanent_uuid: "613d003c41ff4c86a70ce0634c46d578" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 33935 } } I0917 09:19:55.492975 16068 raft_consensus.cc:1184] T 93a32378223c423392af40eb79c3e2b9 P f3112f5bdce74833a3bdec2091deb1a5 [term 2 FOLLOWER]: Refusing update from remote peer 613d003c41ff4c86a70ce0634c46d578: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch) I0917 09:19:55.493191 16306 consensus_queue.cc:984] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3112f5bdce74833a3bdec2091deb1a5" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 36533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0917 09:19:55.493783 16313 raft_consensus.cc:2792] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER bf5537cd74844bf389b64a96222d16f0 (127.13.208.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f3112f5bdce74833a3bdec2091deb1a5" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 36533 } } peers { permanent_uuid: "613d003c41ff4c86a70ce0634c46d578" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 33935 } } } I0917 09:19:55.493914 16068 raft_consensus.cc:2792] T 93a32378223c423392af40eb79c3e2b9 P f3112f5bdce74833a3bdec2091deb1a5 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER bf5537cd74844bf389b64a96222d16f0 (127.13.208.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f3112f5bdce74833a3bdec2091deb1a5" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 36533 } } peers { permanent_uuid: "613d003c41ff4c86a70ce0634c46d578" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 33935 } } } I0917 09:19:55.495793 15848 catalog_manager.cc:4120] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 reported cstate change: config changed from index -1 to 3, VOTER bf5537cd74844bf389b64a96222d16f0 (127.13.208.66) evicted. New cstate: current_term: 2 leader_uuid: "613d003c41ff4c86a70ce0634c46d578" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f3112f5bdce74833a3bdec2091deb1a5" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 36533 } } peers { permanent_uuid: "613d003c41ff4c86a70ce0634c46d578" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 33935 } } } I0917 09:19:55.496402 15848 catalog_manager.cc:3512] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 93a32378223c423392af40eb79c3e2b9 on bf5537cd74844bf389b64a96222d16f0 (127.13.208.66:40967) (TS bf5537cd74844bf389b64a96222d16f0 not found in new config with opid_index 3) I0917 09:19:55.496454 15848 catalog_manager.cc:3739] Sending ChangeConfig:ADD_PEER:VOTER on tablet 93a32378223c423392af40eb79c3e2b9 (attempt 1) I0917 09:19:55.496652 15986 tablet_service.cc:1066] Processing DeleteTablet for tablet 93a32378223c423392af40eb79c3e2b9 with delete_type TABLET_DATA_TOMBSTONED (TS bf5537cd74844bf389b64a96222d16f0 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:37144 I0917 09:19:55.496767 15933 consensus_queue.cc:206] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f3112f5bdce74833a3bdec2091deb1a5" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 36533 } } peers { permanent_uuid: "613d003c41ff4c86a70ce0634c46d578" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 33935 } } peers { permanent_uuid: "bf5537cd74844bf389b64a96222d16f0" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40967 } } I0917 09:19:55.496976 16351 ts_tablet_manager.cc:1440] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0917 09:19:55.497082 16068 raft_consensus.cc:1184] T 93a32378223c423392af40eb79c3e2b9 P f3112f5bdce74833a3bdec2091deb1a5 [term 2 FOLLOWER]: Refusing update from remote peer 613d003c41ff4c86a70ce0634c46d578: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch) W0917 09:19:55.497160 15895 consensus_peers.cc:458] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 -> Peer bf5537cd74844bf389b64a96222d16f0 (127.13.208.66:40967): Couldn't send request to peer bf5537cd74844bf389b64a96222d16f0. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry. I0917 09:19:55.497272 16306 consensus_queue.cc:984] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3112f5bdce74833a3bdec2091deb1a5" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 36533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I0917 09:19:55.497961 16313 raft_consensus.cc:2792] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER bf5537cd74844bf389b64a96222d16f0 (127.13.208.66) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f3112f5bdce74833a3bdec2091deb1a5" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 36533 } } peers { permanent_uuid: "613d003c41ff4c86a70ce0634c46d578" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 33935 } } peers { permanent_uuid: "bf5537cd74844bf389b64a96222d16f0" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40967 } } } I0917 09:19:55.498214 16068 raft_consensus.cc:2792] T 93a32378223c423392af40eb79c3e2b9 P f3112f5bdce74833a3bdec2091deb1a5 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER bf5537cd74844bf389b64a96222d16f0 (127.13.208.66) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f3112f5bdce74833a3bdec2091deb1a5" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 36533 } } peers { permanent_uuid: "613d003c41ff4c86a70ce0634c46d578" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 33935 } } peers { permanent_uuid: "bf5537cd74844bf389b64a96222d16f0" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40967 } } } I0917 09:19:55.499889 16351 ts_tablet_manager.cc:1453] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1 I0917 09:19:55.499917 15833 catalog_manager.cc:3654] ChangeConfig:ADD_PEER:VOTER RPC for tablet 93a32378223c423392af40eb79c3e2b9 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1) I0917 09:19:55.499954 16351 log.cc:990] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0: Deleting WAL directory at /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1568711978778636-14145/minicluster-data/ts-1-root/wals/93a32378223c423392af40eb79c3e2b9 I0917 09:19:55.500339 15846 catalog_manager.cc:4120] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 reported cstate change: config changed from index 3 to 4, VOTER bf5537cd74844bf389b64a96222d16f0 (127.13.208.66) added. New cstate: current_term: 2 leader_uuid: "613d003c41ff4c86a70ce0634c46d578" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f3112f5bdce74833a3bdec2091deb1a5" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 36533 } } peers { permanent_uuid: "613d003c41ff4c86a70ce0634c46d578" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 33935 } } peers { permanent_uuid: "bf5537cd74844bf389b64a96222d16f0" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40967 } } } I0917 09:19:55.500345 15831 catalog_manager.cc:3490] TS bf5537cd74844bf389b64a96222d16f0 (127.13.208.66:40967): tablet 93a32378223c423392af40eb79c3e2b9 (table test-workload [id=6ce6072f86684f7ab7fc2b10ba4b0948]) successfully deleted I0917 09:19:55.891438 16352 ts_tablet_manager.cc:708] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0: Initiating tablet copy from peer 613d003c41ff4c86a70ce0634c46d578 (127.13.208.65:33935) I0917 09:19:55.891503 16352 tablet_copy_client.cc:241] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0: tablet copy: Beginning tablet copy session from remote peer at address 127.13.208.65:33935 I0917 09:19:55.891743 15943 tablet_copy_service.cc:135] P 613d003c41ff4c86a70ce0634c46d578: Received BeginTabletCopySession request for tablet 93a32378223c423392af40eb79c3e2b9 from peer bf5537cd74844bf389b64a96222d16f0 ({username='slave'} at 127.0.0.1:52914) I0917 09:19:55.891767 15943 tablet_copy_service.cc:156] P 613d003c41ff4c86a70ce0634c46d578: Beginning new tablet copy session on tablet 93a32378223c423392af40eb79c3e2b9 from peer bf5537cd74844bf389b64a96222d16f0 at {username='slave'} at 127.0.0.1:52914: session id = bf5537cd74844bf389b64a96222d16f0-93a32378223c423392af40eb79c3e2b9 I0917 09:19:55.892283 15943 tablet_copy_source_session.cc:202] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578: Tablet Copy: opened 0 blocks and 1 log segments I0917 09:19:55.892554 16352 ts_tablet_manager.cc:1440] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0: Deleting tablet data with delete state TABLET_DATA_COPYING I0917 09:19:55.896600 16352 ts_tablet_manager.cc:1453] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1 I0917 09:19:55.896672 16352 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 93a32378223c423392af40eb79c3e2b9. 1 dirs total, 0 dirs full, 0 dirs failed I0917 09:19:55.897289 16352 tablet_copy_client.cc:579] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0: tablet copy: Starting download of 0 data blocks... I0917 09:19:55.897351 16352 tablet_copy_client.cc:542] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0: tablet copy: Starting download of 1 WAL segments... I0917 09:19:55.897981 16352 tablet_copy_client.cc:418] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0917 09:19:55.898526 16352 tablet_bootstrap.cc:439] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0: Bootstrap starting. I0917 09:19:55.899513 16352 tablet_bootstrap.cc:439] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0917 09:19:55.899631 16352 tablet_bootstrap.cc:439] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0: Bootstrap complete. I0917 09:19:55.899698 16352 ts_tablet_manager.cc:1085] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I0917 09:19:55.899857 16352 raft_consensus.cc:340] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f3112f5bdce74833a3bdec2091deb1a5" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 36533 } } peers { permanent_uuid: "613d003c41ff4c86a70ce0634c46d578" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 33935 } } peers { permanent_uuid: "bf5537cd74844bf389b64a96222d16f0" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40967 } } I0917 09:19:55.899874 16352 raft_consensus.cc:705] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf5537cd74844bf389b64a96222d16f0, State: Initialized, Role: FOLLOWER I0917 09:19:55.899926 16352 consensus_queue.cc:229] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f3112f5bdce74833a3bdec2091deb1a5" member_type: VOTER last_known_addr { host: "127.13.208.67" port: 36533 } } peers { permanent_uuid: "613d003c41ff4c86a70ce0634c46d578" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 33935 } } peers { permanent_uuid: "bf5537cd74844bf389b64a96222d16f0" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 40967 } } I0917 09:19:55.900032 16352 ts_tablet_manager.cc:1113] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s I0917 09:19:55.900205 15943 tablet_copy_service.cc:325] P 613d003c41ff4c86a70ce0634c46d578: Request end of tablet copy session bf5537cd74844bf389b64a96222d16f0-93a32378223c423392af40eb79c3e2b9 received from {username='slave'} at 127.0.0.1:52914 I0917 09:19:55.900228 15943 tablet_copy_service.cc:413] P 613d003c41ff4c86a70ce0634c46d578: ending tablet copy session bf5537cd74844bf389b64a96222d16f0-93a32378223c423392af40eb79c3e2b9 on tablet 93a32378223c423392af40eb79c3e2b9 with peer bf5537cd74844bf389b64a96222d16f0 I0917 09:19:56.374450 16001 raft_consensus.cc:1126] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.3->[2.4-2.4] Dedup: 2.4->[] Master Summary UUID | Address | Status ----------------------------------+----------------------+--------- ebc3e15792a94db9b35e13b731cdd6f0 | 127.13.208.126:45123 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+---------------------+---------+---------- 613d003c41ff4c86a70ce0634c46d578 | 127.13.208.65:33935 | HEALTHY | bf5537cd74844bf389b64a96222d16f0 | 127.13.208.66:40967 | HEALTHY | f3112f5bdce74833a3bdec2091deb1a5 | 127.13.208.67:36533 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+-------+-----------------------+------------------------- raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 4 server(s) checked Tablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+---------+---------------+---------+------------+------------------+------------- test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0917 09:19:56.409325 14145 log_verifier.cc:121] Checking tablet 93a32378223c423392af40eb79c3e2b9 I0917 09:19:56.420786 14145 log_verifier.cc:172] Verified matching terms for 4 ops in tablet 93a32378223c423392af40eb79c3e2b9 I0917 09:19:56.421989 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:56.425097 14145 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0917 09:19:56.425137 14145 tablet_replica.cc:273] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578: stopping tablet replica I0917 09:19:56.425174 14145 raft_consensus.cc:2147] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 [term 2 LEADER]: Raft consensus shutting down. I0917 09:19:56.425236 14145 raft_consensus.cc:2174] T 93a32378223c423392af40eb79c3e2b9 P 613d003c41ff4c86a70ce0634c46d578 [term 2 FOLLOWER]: Raft consensus is shut down! I0917 09:19:56.437773 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:56.437808 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:56.437835 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:56.438568 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:56.442188 14145 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0917 09:19:56.442221 14145 tablet_replica.cc:273] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0: stopping tablet replica I0917 09:19:56.442251 14145 raft_consensus.cc:2147] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0 [term 2 FOLLOWER]: Raft consensus shutting down. I0917 09:19:56.442270 14145 raft_consensus.cc:2174] T 93a32378223c423392af40eb79c3e2b9 P bf5537cd74844bf389b64a96222d16f0 [term 2 FOLLOWER]: Raft consensus is shut down! I0917 09:19:56.454597 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:56.454634 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:56.454689 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:56.455811 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:56.459394 14145 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0917 09:19:56.459429 14145 tablet_replica.cc:273] T 93a32378223c423392af40eb79c3e2b9 P f3112f5bdce74833a3bdec2091deb1a5: stopping tablet replica I0917 09:19:56.459457 14145 raft_consensus.cc:2147] T 93a32378223c423392af40eb79c3e2b9 P f3112f5bdce74833a3bdec2091deb1a5 [term 2 FOLLOWER]: Raft consensus shutting down. I0917 09:19:56.459475 14145 raft_consensus.cc:2174] T 93a32378223c423392af40eb79c3e2b9 P f3112f5bdce74833a3bdec2091deb1a5 [term 2 FOLLOWER]: Raft consensus is shut down! I0917 09:19:56.471618 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:56.471652 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:56.471696 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:56.472599 14145 master.cc:247] Master@127.13.208.126:45123 shutting down... I0917 09:19:56.475747 14145 raft_consensus.cc:2147] T 00000000000000000000000000000000 P ebc3e15792a94db9b35e13b731cdd6f0 [term 1 LEADER]: Raft consensus shutting down. I0917 09:19:56.475790 14145 raft_consensus.cc:2174] T 00000000000000000000000000000000 P ebc3e15792a94db9b35e13b731cdd6f0 [term 1 FOLLOWER]: Raft consensus is shut down! I0917 09:19:56.475806 14145 tablet_replica.cc:273] T 00000000000000000000000000000000 P ebc3e15792a94db9b35e13b731cdd6f0: stopping tablet replica I0917 09:19:56.487339 14145 master.cc:258] Master@127.13.208.126:45123 shutdown complete. [ OK ] FailedTabletsAreReplacedITest.OneReplica/0 (4332 ms) [ RUN ] FailedTabletsAreReplacedITest.OneReplica/1 I0917 09:19:56.491190 14145 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.13.208.126:46829 I0917 09:19:56.492025 14145 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 30804us I0917 09:19:56.492081 14145 fs_manager.cc:263] Metadata directory not provided I0917 09:19:56.492106 14145 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0917 09:19:56.492141 14145 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0917 09:19:56.492149 14145 server_base.cc:453] Attempting to create new FS layout instead I0917 09:19:56.492381 14145 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568711978778636-14145/minicluster-data/master-0-root/instance: uuid: "e3206a60b080419f84f498250a83fabf" format_stamp: "Formatted at 2019-09-17 09:19:56 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:56.493491 14145 fs_manager.cc:503] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s I0917 09:19:56.493726 14145 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0917 09:19:56.494323 14145 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:56.494360 14145 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568711978778636-14145/minicluster-data/master-0-root uuid: "e3206a60b080419f84f498250a83fabf" format_stamp: "Formatted at 2019-09-17 09:19:56 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:56.494385 14145 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568711978778636-14145/minicluster-data/master-0-root metadata directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568711978778636-14145/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568711978778636-14145/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0917 09:19:56.508831 14145 rpc_server.cc:205] RPC server started. Bound to: 127.13.208.126:46829 I0917 09:19:56.509188 14145 webserver.cc:374] Webserver started at http://127.13.208.126:38323/ using document root and password file I0917 09:19:56.509441 16438 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0917 09:19:56.510342 16438 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P e3206a60b080419f84f498250a83fabf: Bootstrap starting. I0917 09:19:56.510546 16438 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P e3206a60b080419f84f498250a83fabf: No blocks or log segments found. Creating new log. I0917 09:19:56.510954 16438 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P e3206a60b080419f84f498250a83fabf: No bootstrap required, opened a new log I0917 09:19:56.511106 16438 raft_consensus.cc:340] T 00000000000000000000000000000000 P e3206a60b080419f84f498250a83fabf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3206a60b080419f84f498250a83fabf" member_type: VOTER } I0917 09:19:56.511121 16438 raft_consensus.cc:366] T 00000000000000000000000000000000 P e3206a60b080419f84f498250a83fabf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0917 09:19:56.511129 16438 raft_consensus.cc:705] T 00000000000000000000000000000000 P e3206a60b080419f84f498250a83fabf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3206a60b080419f84f498250a83fabf, State: Initialized, Role: FOLLOWER I0917 09:19:56.511186 16438 consensus_queue.cc:229] T 00000000000000000000000000000000 P e3206a60b080419f84f498250a83fabf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3206a60b080419f84f498250a83fabf" member_type: VOTER } I0917 09:19:56.511198 16438 raft_consensus.cc:380] T 00000000000000000000000000000000 P e3206a60b080419f84f498250a83fabf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0917 09:19:56.511205 16438 raft_consensus.cc:472] T 00000000000000000000000000000000 P e3206a60b080419f84f498250a83fabf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0917 09:19:56.511214 16438 raft_consensus.cc:2890] T 00000000000000000000000000000000 P e3206a60b080419f84f498250a83fabf [term 0 FOLLOWER]: Advancing to term 1 I0917 09:19:56.612356 16438 raft_consensus.cc:494] T 00000000000000000000000000000000 P e3206a60b080419f84f498250a83fabf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3206a60b080419f84f498250a83fabf" member_type: VOTER } I0917 09:19:56.612412 16438 leader_election.cc:310] T 00000000000000000000000000000000 P e3206a60b080419f84f498250a83fabf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3206a60b080419f84f498250a83fabf; no voters: I0917 09:19:56.612527 16438 leader_election.cc:296] T 00000000000000000000000000000000 P e3206a60b080419f84f498250a83fabf [CANDIDATE]: Term 1 election: Requested vote from peers I0917 09:19:56.612625 16438 sys_catalog.cc:340] T 00000000000000000000000000000000 P e3206a60b080419f84f498250a83fabf [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3206a60b080419f84f498250a83fabf" member_type: VOTER } } I0917 09:19:56.612639 16438 sys_catalog.cc:343] T 00000000000000000000000000000000 P e3206a60b080419f84f498250a83fabf [sys.catalog]: This master's current role is: FOLLOWER I0917 09:19:56.612730 16438 sys_catalog.cc:428] T 00000000000000000000000000000000 P e3206a60b080419f84f498250a83fabf [sys.catalog]: configured and running, proceeding with master startup. I0917 09:19:56.612838 14145 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0917 09:19:56.612932 16440 raft_consensus.cc:2644] T 00000000000000000000000000000000 P e3206a60b080419f84f498250a83fabf [term 1 FOLLOWER]: Leader election won for term 1 I0917 09:19:56.613102 16440 raft_consensus.cc:667] T 00000000000000000000000000000000 P e3206a60b080419f84f498250a83fabf [term 1 LEADER]: Becoming Leader. State: Replica: e3206a60b080419f84f498250a83fabf, State: Running, Role: LEADER I0917 09:19:56.613162 16440 consensus_queue.cc:206] T 00000000000000000000000000000000 P e3206a60b080419f84f498250a83fabf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3206a60b080419f84f498250a83fabf" member_type: VOTER } I0917 09:19:56.613515 16441 sys_catalog.cc:340] T 00000000000000000000000000000000 P e3206a60b080419f84f498250a83fabf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3206a60b080419f84f498250a83fabf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3206a60b080419f84f498250a83fabf" member_type: VOTER } } W0917 09:19:56.613586 16442 catalog_manager.cc:1144] T 00000000000000000000000000000000 P e3206a60b080419f84f498250a83fabf: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0917 09:19:56.613644 16442 catalog_manager.cc:627] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0917 09:19:56.613587 16441 sys_catalog.cc:343] T 00000000000000000000000000000000 P e3206a60b080419f84f498250a83fabf [sys.catalog]: This master's current role is: LEADER I0917 09:19:56.613950 16443 sys_catalog.cc:340] T 00000000000000000000000000000000 P e3206a60b080419f84f498250a83fabf [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3206a60b080419f84f498250a83fabf. Latest consensus state: current_term: 1 leader_uuid: "e3206a60b080419f84f498250a83fabf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3206a60b080419f84f498250a83fabf" member_type: VOTER } } I0917 09:19:56.614037 16443 sys_catalog.cc:343] T 00000000000000000000000000000000 P e3206a60b080419f84f498250a83fabf [sys.catalog]: This master's current role is: LEADER I0917 09:19:56.614046 16445 catalog_manager.cc:1086] Loading table and tablet metadata into memory... I0917 09:19:56.614167 16445 catalog_manager.cc:1097] Initializing Kudu internal certificate authority... I0917 09:19:56.640807 16445 catalog_manager.cc:981] Generated new certificate authority record I0917 09:19:56.641369 16445 catalog_manager.cc:1106] Loading token signing keys... I0917 09:19:56.653339 16445 catalog_manager.cc:4435] T 00000000000000000000000000000000 P e3206a60b080419f84f498250a83fabf: Generated new TSK 0 I0917 09:19:56.678352 14145 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 30804us I0917 09:19:56.678424 14145 fs_manager.cc:263] Metadata directory not provided I0917 09:19:56.678442 14145 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0917 09:19:56.678467 14145 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0917 09:19:56.678473 14145 server_base.cc:453] Attempting to create new FS layout instead I0917 09:19:56.678730 14145 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568711978778636-14145/minicluster-data/ts-0-root/instance: uuid: "2dd78402759d4877b2991ce5ad9abd06" format_stamp: "Formatted at 2019-09-17 09:19:56 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:56.679901 14145 fs_manager.cc:503] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s I0917 09:19:56.680188 14145 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0917 09:19:56.680922 14145 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:56.680965 14145 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568711978778636-14145/minicluster-data/ts-0-root uuid: "2dd78402759d4877b2991ce5ad9abd06" format_stamp: "Formatted at 2019-09-17 09:19:56 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:56.680985 14145 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568711978778636-14145/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568711978778636-14145/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568711978778636-14145/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0917 09:19:56.692255 14145 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0917 09:19:56.692281 14145 ts_tablet_manager.cc:403] Registered 0 tablets I0917 09:19:56.695039 14145 rpc_server.cc:205] RPC server started. Bound to: 127.13.208.65:39567 I0917 09:19:56.695309 14145 webserver.cc:374] Webserver started at http://127.13.208.65:39691/ using document root and password file I0917 09:19:56.697089 14145 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 30804us I0917 09:19:56.697165 14145 fs_manager.cc:263] Metadata directory not provided I0917 09:19:56.697199 14145 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0917 09:19:56.697229 14145 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0917 09:19:56.697240 14145 server_base.cc:453] Attempting to create new FS layout instead I0917 09:19:56.697605 14145 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568711978778636-14145/minicluster-data/ts-1-root/instance: uuid: "7a66fc8acece469db093d36cf50b448e" format_stamp: "Formatted at 2019-09-17 09:19:56 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:56.699000 16512 heartbeater.cc:322] Connected to a master server at 127.13.208.126:46829 I0917 09:19:56.699050 16512 heartbeater.cc:406] Registering TS with master... I0917 09:19:56.699108 16512 heartbeater.cc:455] Master 127.13.208.126:46829 requested a full tablet report, sending... I0917 09:19:56.699158 14145 fs_manager.cc:503] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s I0917 09:19:56.699429 14145 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0917 09:19:56.699540 16403 ts_manager.cc:161] Registered new tserver with Master: 2dd78402759d4877b2991ce5ad9abd06 (127.13.208.65:39567) I0917 09:19:56.700428 14145 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:56.700472 14145 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568711978778636-14145/minicluster-data/ts-1-root uuid: "7a66fc8acece469db093d36cf50b448e" format_stamp: "Formatted at 2019-09-17 09:19:56 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:56.700479 16403 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43028 I0917 09:19:56.700511 14145 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568711978778636-14145/minicluster-data/ts-1-root metadata directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568711978778636-14145/minicluster-data/ts-1-root 1 data directories: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568711978778636-14145/minicluster-data/ts-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0917 09:19:56.714946 14145 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0917 09:19:56.714977 14145 ts_tablet_manager.cc:403] Registered 0 tablets I0917 09:19:56.717118 14145 rpc_server.cc:205] RPC server started. Bound to: 127.13.208.66:46523 I0917 09:19:56.717330 14145 webserver.cc:374] Webserver started at http://127.13.208.66:40697/ using document root and password file I0917 09:19:56.719882 14145 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 30804us I0917 09:19:56.719964 14145 fs_manager.cc:263] Metadata directory not provided I0917 09:19:56.720023 14145 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0917 09:19:56.720062 14145 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0917 09:19:56.720075 14145 server_base.cc:453] Attempting to create new FS layout instead I0917 09:19:56.720327 14145 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568711978778636-14145/minicluster-data/ts-2-root/instance: uuid: "156e622c1244482eb132ee4e55767682" format_stamp: "Formatted at 2019-09-17 09:19:56 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:56.721385 16580 heartbeater.cc:322] Connected to a master server at 127.13.208.126:46829 I0917 09:19:56.721417 16580 heartbeater.cc:406] Registering TS with master... I0917 09:19:56.721462 16580 heartbeater.cc:455] Master 127.13.208.126:46829 requested a full tablet report, sending... I0917 09:19:56.721657 16403 ts_manager.cc:161] Registered new tserver with Master: 7a66fc8acece469db093d36cf50b448e (127.13.208.66:46523) I0917 09:19:56.722059 16403 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43030 I0917 09:19:56.787633 14145 fs_manager.cc:503] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s I0917 09:19:56.787943 14145 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0917 09:19:56.789321 14145 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:56.789425 14145 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568711978778636-14145/minicluster-data/ts-2-root uuid: "156e622c1244482eb132ee4e55767682" format_stamp: "Formatted at 2019-09-17 09:19:56 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:56.789453 14145 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568711978778636-14145/minicluster-data/ts-2-root metadata directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568711978778636-14145/minicluster-data/ts-2-root 1 data directories: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568711978778636-14145/minicluster-data/ts-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0917 09:19:56.818033 14145 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0917 09:19:56.818060 14145 ts_tablet_manager.cc:403] Registered 0 tablets I0917 09:19:56.820156 14145 rpc_server.cc:205] RPC server started. Bound to: 127.13.208.67:34059 I0917 09:19:56.820436 14145 webserver.cc:374] Webserver started at http://127.13.208.67:37147/ using document root and password file I0917 09:19:56.823328 16647 heartbeater.cc:322] Connected to a master server at 127.13.208.126:46829 I0917 09:19:56.823360 16647 heartbeater.cc:406] Registering TS with master... I0917 09:19:56.823406 16647 heartbeater.cc:455] Master 127.13.208.126:46829 requested a full tablet report, sending... I0917 09:19:56.823580 16403 ts_manager.cc:161] Registered new tserver with Master: 156e622c1244482eb132ee4e55767682 (127.13.208.67:34059) I0917 09:19:56.823761 14145 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 30804us I0917 09:19:56.823837 14145 fs_manager.cc:263] Metadata directory not provided I0917 09:19:56.823868 14145 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0917 09:19:56.823902 14145 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0917 09:19:56.823912 14145 server_base.cc:453] Attempting to create new FS layout instead I0917 09:19:56.824082 16403 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43032 I0917 09:19:56.824156 14145 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568711978778636-14145/minicluster-data/ts-3-root/instance: uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" format_stamp: "Formatted at 2019-09-17 09:19:56 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:56.826218 14145 fs_manager.cc:503] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s I0917 09:19:56.826510 14145 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0917 09:19:56.828025 14145 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0917 09:19:56.828065 14145 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568711978778636-14145/minicluster-data/ts-3-root uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" format_stamp: "Formatted at 2019-09-17 09:19:56 on dist-test-slave-dist-test-slave-w6nz" I0917 09:19:56.828097 14145 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568711978778636-14145/minicluster-data/ts-3-root metadata directory: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568711978778636-14145/minicluster-data/ts-3-root 1 data directories: /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568711978778636-14145/minicluster-data/ts-3-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0917 09:19:56.850250 14145 ts_tablet_manager.cc:383] Loaded tablet metadata (0 total tablets, 0 live tablets) I0917 09:19:56.850277 14145 ts_tablet_manager.cc:403] Registered 0 tablets I0917 09:19:56.852217 14145 rpc_server.cc:205] RPC server started. Bound to: 127.13.208.68:37535 I0917 09:19:56.852509 14145 webserver.cc:374] Webserver started at http://127.13.208.68:42019/ using document root and password file I0917 09:19:56.855826 16714 heartbeater.cc:322] Connected to a master server at 127.13.208.126:46829 I0917 09:19:56.855885 16714 heartbeater.cc:406] Registering TS with master... I0917 09:19:56.855950 16714 heartbeater.cc:455] Master 127.13.208.126:46829 requested a full tablet report, sending... I0917 09:19:56.856187 16403 ts_manager.cc:161] Registered new tserver with Master: ce9933ca3a5c4f5ab9f3a41857b7fbb2 (127.13.208.68:37535) I0917 09:19:56.856613 16403 master_service.cc:276] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43034 I0917 09:19:56.856974 14145 internal_mini_cluster.cc:328] 4 TS(s) registered with all masters after 0.004339925s I0917 09:19:56.861207 14145 test_util.cc:212] Using random seed: 961800980 I0917 09:19:56.865377 16403 catalog_manager.cc:1440] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43038: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0917 09:19:56.871901 16547 tablet_service.cc:1022] Processing CreateTablet for tablet 5e9d0599e0df416e97e63d608db6f0ab (table=test-workload [id=1deae85f7ecc48d3ae00be11b1a14d9c]), partition=RANGE (key) PARTITION UNBOUNDED I0917 09:19:56.872023 16547 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 5e9d0599e0df416e97e63d608db6f0ab. 1 dirs total, 0 dirs full, 0 dirs failed I0917 09:19:56.873672 16681 tablet_service.cc:1022] Processing CreateTablet for tablet 5e9d0599e0df416e97e63d608db6f0ab (table=test-workload [id=1deae85f7ecc48d3ae00be11b1a14d9c]), partition=RANGE (key) PARTITION UNBOUNDED I0917 09:19:56.873875 16681 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 5e9d0599e0df416e97e63d608db6f0ab. 1 dirs total, 0 dirs full, 0 dirs failed I0917 09:19:56.874122 16473 tablet_service.cc:1022] Processing CreateTablet for tablet 5e9d0599e0df416e97e63d608db6f0ab (table=test-workload [id=1deae85f7ecc48d3ae00be11b1a14d9c]), partition=RANGE (key) PARTITION UNBOUNDED I0917 09:19:56.874222 16473 data_dirs.cc:939] Could only allocate 1 dirs of requested 3 for tablet 5e9d0599e0df416e97e63d608db6f0ab. 1 dirs total, 0 dirs full, 0 dirs failed I0917 09:19:56.874817 16734 tablet_bootstrap.cc:439] T 5e9d0599e0df416e97e63d608db6f0ab P 7a66fc8acece469db093d36cf50b448e: Bootstrap starting. I0917 09:19:56.875350 16734 tablet_bootstrap.cc:590] T 5e9d0599e0df416e97e63d608db6f0ab P 7a66fc8acece469db093d36cf50b448e: No blocks or log segments found. Creating new log. I0917 09:19:56.876070 16734 tablet_bootstrap.cc:439] T 5e9d0599e0df416e97e63d608db6f0ab P 7a66fc8acece469db093d36cf50b448e: No bootstrap required, opened a new log I0917 09:19:56.876180 16734 ts_tablet_manager.cc:1085] T 5e9d0599e0df416e97e63d608db6f0ab P 7a66fc8acece469db093d36cf50b448e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I0917 09:19:56.876535 16734 raft_consensus.cc:340] T 5e9d0599e0df416e97e63d608db6f0ab P 7a66fc8acece469db093d36cf50b448e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" member_type: VOTER last_known_addr { host: "127.13.208.68" port: 37535 } } peers { permanent_uuid: "7a66fc8acece469db093d36cf50b448e" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 46523 } } peers { permanent_uuid: "2dd78402759d4877b2991ce5ad9abd06" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39567 } } I0917 09:19:56.876663 16734 raft_consensus.cc:366] T 5e9d0599e0df416e97e63d608db6f0ab P 7a66fc8acece469db093d36cf50b448e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0917 09:19:56.876714 16734 raft_consensus.cc:705] T 5e9d0599e0df416e97e63d608db6f0ab P 7a66fc8acece469db093d36cf50b448e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a66fc8acece469db093d36cf50b448e, State: Initialized, Role: FOLLOWER I0917 09:19:56.877034 16734 consensus_queue.cc:229] T 5e9d0599e0df416e97e63d608db6f0ab P 7a66fc8acece469db093d36cf50b448e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" member_type: VOTER last_known_addr { host: "127.13.208.68" port: 37535 } } peers { permanent_uuid: "7a66fc8acece469db093d36cf50b448e" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 46523 } } peers { permanent_uuid: "2dd78402759d4877b2991ce5ad9abd06" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39567 } } I0917 09:19:56.877293 16580 heartbeater.cc:447] Master 127.13.208.126:46829 was elected leader, sending a full tablet report... I0917 09:19:56.877429 16734 ts_tablet_manager.cc:1113] T 5e9d0599e0df416e97e63d608db6f0ab P 7a66fc8acece469db093d36cf50b448e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0917 09:19:56.877537 16737 tablet_bootstrap.cc:439] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2: Bootstrap starting. I0917 09:19:56.877997 16737 tablet_bootstrap.cc:590] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2: No blocks or log segments found. Creating new log. I0917 09:19:56.878021 16738 tablet_bootstrap.cc:439] T 5e9d0599e0df416e97e63d608db6f0ab P 2dd78402759d4877b2991ce5ad9abd06: Bootstrap starting. I0917 09:19:56.878573 16738 tablet_bootstrap.cc:590] T 5e9d0599e0df416e97e63d608db6f0ab P 2dd78402759d4877b2991ce5ad9abd06: No blocks or log segments found. Creating new log. I0917 09:19:56.879043 16737 tablet_bootstrap.cc:439] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2: No bootstrap required, opened a new log I0917 09:19:56.879093 16738 tablet_bootstrap.cc:439] T 5e9d0599e0df416e97e63d608db6f0ab P 2dd78402759d4877b2991ce5ad9abd06: No bootstrap required, opened a new log I0917 09:19:56.879099 16737 ts_tablet_manager.cc:1085] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I0917 09:19:56.879137 16738 ts_tablet_manager.cc:1085] T 5e9d0599e0df416e97e63d608db6f0ab P 2dd78402759d4877b2991ce5ad9abd06: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I0917 09:19:56.879377 16738 raft_consensus.cc:340] T 5e9d0599e0df416e97e63d608db6f0ab P 2dd78402759d4877b2991ce5ad9abd06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" member_type: VOTER last_known_addr { host: "127.13.208.68" port: 37535 } } peers { permanent_uuid: "7a66fc8acece469db093d36cf50b448e" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 46523 } } peers { permanent_uuid: "2dd78402759d4877b2991ce5ad9abd06" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39567 } } I0917 09:19:56.879384 16737 raft_consensus.cc:340] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" member_type: VOTER last_known_addr { host: "127.13.208.68" port: 37535 } } peers { permanent_uuid: "7a66fc8acece469db093d36cf50b448e" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 46523 } } peers { permanent_uuid: "2dd78402759d4877b2991ce5ad9abd06" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39567 } } I0917 09:19:56.879396 16738 raft_consensus.cc:366] T 5e9d0599e0df416e97e63d608db6f0ab P 2dd78402759d4877b2991ce5ad9abd06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0917 09:19:56.879406 16737 raft_consensus.cc:366] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0917 09:19:56.879408 16738 raft_consensus.cc:705] T 5e9d0599e0df416e97e63d608db6f0ab P 2dd78402759d4877b2991ce5ad9abd06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dd78402759d4877b2991ce5ad9abd06, State: Initialized, Role: FOLLOWER I0917 09:19:56.879422 16737 raft_consensus.cc:705] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce9933ca3a5c4f5ab9f3a41857b7fbb2, State: Initialized, Role: FOLLOWER I0917 09:19:56.879545 16737 consensus_queue.cc:229] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" member_type: VOTER last_known_addr { host: "127.13.208.68" port: 37535 } } peers { permanent_uuid: "7a66fc8acece469db093d36cf50b448e" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 46523 } } peers { permanent_uuid: "2dd78402759d4877b2991ce5ad9abd06" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39567 } } I0917 09:19:56.879556 16738 consensus_queue.cc:229] T 5e9d0599e0df416e97e63d608db6f0ab P 2dd78402759d4877b2991ce5ad9abd06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" member_type: VOTER last_known_addr { host: "127.13.208.68" port: 37535 } } peers { permanent_uuid: "7a66fc8acece469db093d36cf50b448e" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 46523 } } peers { permanent_uuid: "2dd78402759d4877b2991ce5ad9abd06" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39567 } } I0917 09:19:56.879741 16737 ts_tablet_manager.cc:1113] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0917 09:19:56.879747 16738 ts_tablet_manager.cc:1113] T 5e9d0599e0df416e97e63d608db6f0ab P 2dd78402759d4877b2991ce5ad9abd06: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0917 09:19:56.879765 16512 heartbeater.cc:447] Master 127.13.208.126:46829 was elected leader, sending a full tablet report... I0917 09:19:56.879788 16714 heartbeater.cc:447] Master 127.13.208.126:46829 was elected leader, sending a full tablet report... I0917 09:19:57.037299 16741 raft_consensus.cc:472] T 5e9d0599e0df416e97e63d608db6f0ab P 2dd78402759d4877b2991ce5ad9abd06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0917 09:19:57.037475 16741 raft_consensus.cc:494] T 5e9d0599e0df416e97e63d608db6f0ab P 2dd78402759d4877b2991ce5ad9abd06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" member_type: VOTER last_known_addr { host: "127.13.208.68" port: 37535 } } peers { permanent_uuid: "7a66fc8acece469db093d36cf50b448e" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 46523 } } peers { permanent_uuid: "2dd78402759d4877b2991ce5ad9abd06" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39567 } } I0917 09:19:57.037617 16741 leader_election.cc:296] T 5e9d0599e0df416e97e63d608db6f0ab P 2dd78402759d4877b2991ce5ad9abd06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce9933ca3a5c4f5ab9f3a41857b7fbb2 (127.13.208.68:37535), 7a66fc8acece469db093d36cf50b448e (127.13.208.66:46523) I0917 09:19:57.042389 16557 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5e9d0599e0df416e97e63d608db6f0ab" candidate_uuid: "2dd78402759d4877b2991ce5ad9abd06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a66fc8acece469db093d36cf50b448e" is_pre_election: true I0917 09:19:57.042455 16557 raft_consensus.cc:2350] T 5e9d0599e0df416e97e63d608db6f0ab P 7a66fc8acece469db093d36cf50b448e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dd78402759d4877b2991ce5ad9abd06 in term 0. I0917 09:19:57.042729 16453 leader_election.cc:310] T 5e9d0599e0df416e97e63d608db6f0ab P 2dd78402759d4877b2991ce5ad9abd06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dd78402759d4877b2991ce5ad9abd06, 7a66fc8acece469db093d36cf50b448e; no voters: I0917 09:19:57.042819 16741 raft_consensus.cc:2644] T 5e9d0599e0df416e97e63d608db6f0ab P 2dd78402759d4877b2991ce5ad9abd06 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0917 09:19:57.042866 16741 raft_consensus.cc:472] T 5e9d0599e0df416e97e63d608db6f0ab P 2dd78402759d4877b2991ce5ad9abd06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0917 09:19:57.042898 16741 raft_consensus.cc:2890] T 5e9d0599e0df416e97e63d608db6f0ab P 2dd78402759d4877b2991ce5ad9abd06 [term 0 FOLLOWER]: Advancing to term 1 I0917 09:19:57.042999 16691 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5e9d0599e0df416e97e63d608db6f0ab" candidate_uuid: "2dd78402759d4877b2991ce5ad9abd06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" is_pre_election: true I0917 09:19:57.043036 16691 raft_consensus.cc:2350] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dd78402759d4877b2991ce5ad9abd06 in term 0. I0917 09:19:57.043849 16741 raft_consensus.cc:494] T 5e9d0599e0df416e97e63d608db6f0ab P 2dd78402759d4877b2991ce5ad9abd06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" member_type: VOTER last_known_addr { host: "127.13.208.68" port: 37535 } } peers { permanent_uuid: "7a66fc8acece469db093d36cf50b448e" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 46523 } } peers { permanent_uuid: "2dd78402759d4877b2991ce5ad9abd06" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39567 } } I0917 09:19:57.044189 16557 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5e9d0599e0df416e97e63d608db6f0ab" candidate_uuid: "2dd78402759d4877b2991ce5ad9abd06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a66fc8acece469db093d36cf50b448e" I0917 09:19:57.044216 16557 raft_consensus.cc:2890] T 5e9d0599e0df416e97e63d608db6f0ab P 7a66fc8acece469db093d36cf50b448e [term 0 FOLLOWER]: Advancing to term 1 I0917 09:19:57.044317 16691 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5e9d0599e0df416e97e63d608db6f0ab" candidate_uuid: "2dd78402759d4877b2991ce5ad9abd06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" I0917 09:19:57.044320 16741 leader_election.cc:296] T 5e9d0599e0df416e97e63d608db6f0ab P 2dd78402759d4877b2991ce5ad9abd06 [CANDIDATE]: Term 1 election: Requested vote from peers ce9933ca3a5c4f5ab9f3a41857b7fbb2 (127.13.208.68:37535), 7a66fc8acece469db093d36cf50b448e (127.13.208.66:46523) I0917 09:19:57.044373 16691 raft_consensus.cc:2890] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [term 0 FOLLOWER]: Advancing to term 1 I0917 09:19:57.045276 16557 raft_consensus.cc:2350] T 5e9d0599e0df416e97e63d608db6f0ab P 7a66fc8acece469db093d36cf50b448e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dd78402759d4877b2991ce5ad9abd06 in term 1. I0917 09:19:57.045397 16691 raft_consensus.cc:2350] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dd78402759d4877b2991ce5ad9abd06 in term 1. I0917 09:19:57.045428 16453 leader_election.cc:310] T 5e9d0599e0df416e97e63d608db6f0ab P 2dd78402759d4877b2991ce5ad9abd06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dd78402759d4877b2991ce5ad9abd06, 7a66fc8acece469db093d36cf50b448e; no voters: I0917 09:19:57.045500 16741 raft_consensus.cc:2644] T 5e9d0599e0df416e97e63d608db6f0ab P 2dd78402759d4877b2991ce5ad9abd06 [term 1 FOLLOWER]: Leader election won for term 1 I0917 09:19:57.045601 16741 raft_consensus.cc:667] T 5e9d0599e0df416e97e63d608db6f0ab P 2dd78402759d4877b2991ce5ad9abd06 [term 1 LEADER]: Becoming Leader. State: Replica: 2dd78402759d4877b2991ce5ad9abd06, State: Running, Role: LEADER I0917 09:19:57.045702 16741 consensus_queue.cc:206] T 5e9d0599e0df416e97e63d608db6f0ab P 2dd78402759d4877b2991ce5ad9abd06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" member_type: VOTER last_known_addr { host: "127.13.208.68" port: 37535 } } peers { permanent_uuid: "7a66fc8acece469db093d36cf50b448e" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 46523 } } peers { permanent_uuid: "2dd78402759d4877b2991ce5ad9abd06" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39567 } } I0917 09:19:57.046267 16404 catalog_manager.cc:4120] T 5e9d0599e0df416e97e63d608db6f0ab P 2dd78402759d4877b2991ce5ad9abd06 reported cstate change: term changed from 0 to 1, leader changed from to 2dd78402759d4877b2991ce5ad9abd06 (127.13.208.65). New cstate: current_term: 1 leader_uuid: "2dd78402759d4877b2991ce5ad9abd06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" member_type: VOTER last_known_addr { host: "127.13.208.68" port: 37535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a66fc8acece469db093d36cf50b448e" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 46523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dd78402759d4877b2991ce5ad9abd06" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39567 } health_report { overall_health: HEALTHY } } } Master Summary UUID | Address | Status ----------------------------------+----------------------+--------- e3206a60b080419f84f498250a83fabf | 127.13.208.126:46829 | HEALTHY Flag | Value | Tags | Master ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+---------------------+---------+---------- 156e622c1244482eb132ee4e55767682 | 127.13.208.67:34059 | HEALTHY | 2dd78402759d4877b2991ce5ad9abd06 | 127.13.208.65:39567 | HEALTHY | 7a66fc8acece469db093d36cf50b448e | 127.13.208.66:46523 | HEALTHY | ce9933ca3a5c4f5ab9f3a41857b7fbb2 | 127.13.208.68:37535 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet 5e9d0599e0df416e97e63d608db6f0ab of table 'test-workload' is conflicted: 5 replicas' active configs disagree with the leader master's ce9933ca3a5c4f5ab9f3a41857b7fbb2 (127.13.208.68:37535): RUNNING 7a66fc8acece469db093d36cf50b448e (127.13.208.66:46523): RUNNING 2dd78402759d4877b2991ce5ad9abd06 (127.13.208.65:39567): RUNNING [LEADER] All reported replicas are: A = ce9933ca3a5c4f5ab9f3a41857b7fbb2 B = 7a66fc8acece469db093d36cf50b448e C = 2dd78402759d4877b2991ce5ad9abd06 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C | 1 | -1 | Yes B | A B C | 1 | -1 | Yes C | A B C* | 1 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 0 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+---------------------+--------------- Small | 156e622c1244482eb132ee4e55767682 | 127.13.208.67:34059 | 0 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0917 09:19:57.076311 14145 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors Master Summary UUID | Address | Status ----------------------------------+----------------------+--------- e3206a60b080419f84f498250a83fabf | 127.13.208.126:46829 | HEALTHY Flag | Value | Tags | Master ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+---------------------+---------+---------- 156e622c1244482eb132ee4e55767682 | 127.13.208.67:34059 | HEALTHY | 2dd78402759d4877b2991ce5ad9abd06 | 127.13.208.65:39567 | HEALTHY | 7a66fc8acece469db093d36cf50b448e | 127.13.208.66:46523 | HEALTHY | ce9933ca3a5c4f5ab9f3a41857b7fbb2 | 127.13.208.68:37535 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet 5e9d0599e0df416e97e63d608db6f0ab of table 'test-workload' is conflicted: 5 replicas' active configs disagree with the leader master's ce9933ca3a5c4f5ab9f3a41857b7fbb2 (127.13.208.68:37535): RUNNING 7a66fc8acece469db093d36cf50b448e (127.13.208.66:46523): RUNNING 2dd78402759d4877b2991ce5ad9abd06 (127.13.208.65:39567): RUNNING [LEADER] All reported replicas are: A = ce9933ca3a5c4f5ab9f3a41857b7fbb2 B = 7a66fc8acece469db093d36cf50b448e C = 2dd78402759d4877b2991ce5ad9abd06 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C | 1 | -1 | Yes B | A B C | 1 | -1 | Yes C | A B C* | 1 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 0 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+---------------------+--------------- Small | 156e622c1244482eb132ee4e55767682 | 127.13.208.67:34059 | 0 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0917 09:19:57.254976 14145 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0917 09:19:57.464681 16741 consensus_queue.cc:984] T 5e9d0599e0df416e97e63d608db6f0ab P 2dd78402759d4877b2991ce5ad9abd06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" member_type: VOTER last_known_addr { host: "127.13.208.68" port: 37535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0917 09:19:57.466267 16741 consensus_queue.cc:984] T 5e9d0599e0df416e97e63d608db6f0ab P 2dd78402759d4877b2991ce5ad9abd06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a66fc8acece469db093d36cf50b448e" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 46523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s Master Summary UUID | Address | Status ----------------------------------+----------------------+--------- e3206a60b080419f84f498250a83fabf | 127.13.208.126:46829 | HEALTHY Flag | Value | Tags | Master ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+---------------------+---------+---------- 156e622c1244482eb132ee4e55767682 | 127.13.208.67:34059 | HEALTHY | 2dd78402759d4877b2991ce5ad9abd06 | 127.13.208.65:39567 | HEALTHY | 7a66fc8acece469db093d36cf50b448e | 127.13.208.66:46523 | HEALTHY | ce9933ca3a5c4f5ab9f3a41857b7fbb2 | 127.13.208.68:37535 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 5 server(s) checked Tablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+---------+---------------+---------+------------+------------------+------------- test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 0 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+---------------------+--------------- Small | 156e622c1244482eb132ee4e55767682 | 127.13.208.67:34059 | 0 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0917 09:19:57.508343 14145 log_verifier.cc:121] Checking tablet 5e9d0599e0df416e97e63d608db6f0ab I0917 09:19:57.521591 14145 log_verifier.cc:172] Verified matching terms for 1 ops in tablet 5e9d0599e0df416e97e63d608db6f0ab I0917 09:19:57.521688 14145 tablet_replica.cc:273] T 5e9d0599e0df416e97e63d608db6f0ab P 2dd78402759d4877b2991ce5ad9abd06: stopping tablet replica I0917 09:19:57.521728 14145 raft_consensus.cc:2147] T 5e9d0599e0df416e97e63d608db6f0ab P 2dd78402759d4877b2991ce5ad9abd06 [term 1 LEADER]: Raft consensus shutting down. I0917 09:19:57.521811 14145 raft_consensus.cc:2174] T 5e9d0599e0df416e97e63d608db6f0ab P 2dd78402759d4877b2991ce5ad9abd06 [term 1 FOLLOWER]: Raft consensus is shut down! Master Summary UUID | Address | Status ----------------------------------+----------------------+--------- e3206a60b080419f84f498250a83fabf | 127.13.208.126:46829 | HEALTHY Flag | Value | Tags | Master ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+---------------------+---------+---------- 156e622c1244482eb132ee4e55767682 | 127.13.208.67:34059 | HEALTHY | 2dd78402759d4877b2991ce5ad9abd06 | 127.13.208.65:39567 | HEALTHY | 7a66fc8acece469db093d36cf50b448e | 127.13.208.66:46523 | HEALTHY | ce9933ca3a5c4f5ab9f3a41857b7fbb2 | 127.13.208.68:37535 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet 5e9d0599e0df416e97e63d608db6f0ab of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING ce9933ca3a5c4f5ab9f3a41857b7fbb2 (127.13.208.68:37535): RUNNING 7a66fc8acece469db093d36cf50b448e (127.13.208.66:46523): RUNNING 2dd78402759d4877b2991ce5ad9abd06 (127.13.208.65:39567): not running [LEADER] State: FAILED Data state: TABLET_DATA_READY Last status: IO error: INJECTED ERROR: tablet failed All reported replicas are: A = ce9933ca3a5c4f5ab9f3a41857b7fbb2 B = 7a66fc8acece469db093d36cf50b448e C = 2dd78402759d4877b2991ce5ad9abd06 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C* | 1 | -1 | Yes B | A B C* | 1 | -1 | Yes C | [config not available] | | | Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 0 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+---------------------+--------------- Small | 156e622c1244482eb132ee4e55767682 | 127.13.208.67:34059 | 0 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0917 09:19:57.552731 14145 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors Master Summary UUID | Address | Status ----------------------------------+----------------------+--------- e3206a60b080419f84f498250a83fabf | 127.13.208.126:46829 | HEALTHY Flag | Value | Tags | Master ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+---------------------+---------+---------- 156e622c1244482eb132ee4e55767682 | 127.13.208.67:34059 | HEALTHY | 2dd78402759d4877b2991ce5ad9abd06 | 127.13.208.65:39567 | HEALTHY | 7a66fc8acece469db093d36cf50b448e | 127.13.208.66:46523 | HEALTHY | ce9933ca3a5c4f5ab9f3a41857b7fbb2 | 127.13.208.68:37535 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet 5e9d0599e0df416e97e63d608db6f0ab of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING ce9933ca3a5c4f5ab9f3a41857b7fbb2 (127.13.208.68:37535): RUNNING 7a66fc8acece469db093d36cf50b448e (127.13.208.66:46523): RUNNING 2dd78402759d4877b2991ce5ad9abd06 (127.13.208.65:39567): not running [LEADER] State: FAILED Data state: TABLET_DATA_READY Last status: IO error: INJECTED ERROR: tablet failed All reported replicas are: A = ce9933ca3a5c4f5ab9f3a41857b7fbb2 B = 7a66fc8acece469db093d36cf50b448e C = 2dd78402759d4877b2991ce5ad9abd06 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C* | 1 | -1 | Yes B | A B C* | 1 | -1 | Yes C | [config not available] | | | Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 0 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+---------------------+--------------- Small | 156e622c1244482eb132ee4e55767682 | 127.13.208.67:34059 | 0 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0917 09:19:57.731995 14145 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0917 09:19:57.824841 16647 heartbeater.cc:447] Master 127.13.208.126:46829 was elected leader, sending a full tablet report... Master Summary UUID | Address | Status ----------------------------------+----------------------+--------- e3206a60b080419f84f498250a83fabf | 127.13.208.126:46829 | HEALTHY Flag | Value | Tags | Master ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+---------------------+---------+---------- 156e622c1244482eb132ee4e55767682 | 127.13.208.67:34059 | HEALTHY | 2dd78402759d4877b2991ce5ad9abd06 | 127.13.208.65:39567 | HEALTHY | 7a66fc8acece469db093d36cf50b448e | 127.13.208.66:46523 | HEALTHY | ce9933ca3a5c4f5ab9f3a41857b7fbb2 | 127.13.208.68:37535 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet 5e9d0599e0df416e97e63d608db6f0ab of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING ce9933ca3a5c4f5ab9f3a41857b7fbb2 (127.13.208.68:37535): RUNNING 7a66fc8acece469db093d36cf50b448e (127.13.208.66:46523): RUNNING 2dd78402759d4877b2991ce5ad9abd06 (127.13.208.65:39567): not running [LEADER] State: FAILED Data state: TABLET_DATA_READY Last status: IO error: INJECTED ERROR: tablet failed All reported replicas are: A = ce9933ca3a5c4f5ab9f3a41857b7fbb2 B = 7a66fc8acece469db093d36cf50b448e C = 2dd78402759d4877b2991ce5ad9abd06 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C* | 1 | -1 | Yes B | A B C* | 1 | -1 | Yes C | [config not available] | | | Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 0 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+---------------------+--------------- Small | 156e622c1244482eb132ee4e55767682 | 127.13.208.67:34059 | 0 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0917 09:19:57.985018 14145 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors Master Summary UUID | Address | Status ----------------------------------+----------------------+--------- e3206a60b080419f84f498250a83fabf | 127.13.208.126:46829 | HEALTHY Flag | Value | Tags | Master ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+---------------------+---------+---------- 156e622c1244482eb132ee4e55767682 | 127.13.208.67:34059 | HEALTHY | 2dd78402759d4877b2991ce5ad9abd06 | 127.13.208.65:39567 | HEALTHY | 7a66fc8acece469db093d36cf50b448e | 127.13.208.66:46523 | HEALTHY | ce9933ca3a5c4f5ab9f3a41857b7fbb2 | 127.13.208.68:37535 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet 5e9d0599e0df416e97e63d608db6f0ab of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING ce9933ca3a5c4f5ab9f3a41857b7fbb2 (127.13.208.68:37535): RUNNING 7a66fc8acece469db093d36cf50b448e (127.13.208.66:46523): RUNNING 2dd78402759d4877b2991ce5ad9abd06 (127.13.208.65:39567): not running [LEADER] State: FAILED Data state: TABLET_DATA_READY Last status: IO error: INJECTED ERROR: tablet failed All reported replicas are: A = ce9933ca3a5c4f5ab9f3a41857b7fbb2 B = 7a66fc8acece469db093d36cf50b448e C = 2dd78402759d4877b2991ce5ad9abd06 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C* | 1 | -1 | Yes B | A B C* | 1 | -1 | Yes C | [config not available] | | | Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 0 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+---------------------+--------------- Small | 156e622c1244482eb132ee4e55767682 | 127.13.208.67:34059 | 0 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0917 09:19:58.350219 14145 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors Master Summary UUID | Address | Status ----------------------------------+----------------------+--------- e3206a60b080419f84f498250a83fabf | 127.13.208.126:46829 | HEALTHY Flag | Value | Tags | Master ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+---------------------+---------+---------- 156e622c1244482eb132ee4e55767682 | 127.13.208.67:34059 | HEALTHY | 2dd78402759d4877b2991ce5ad9abd06 | 127.13.208.65:39567 | HEALTHY | 7a66fc8acece469db093d36cf50b448e | 127.13.208.66:46523 | HEALTHY | ce9933ca3a5c4f5ab9f3a41857b7fbb2 | 127.13.208.68:37535 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet 5e9d0599e0df416e97e63d608db6f0ab of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING ce9933ca3a5c4f5ab9f3a41857b7fbb2 (127.13.208.68:37535): RUNNING 7a66fc8acece469db093d36cf50b448e (127.13.208.66:46523): RUNNING 2dd78402759d4877b2991ce5ad9abd06 (127.13.208.65:39567): not running [LEADER] State: FAILED Data state: TABLET_DATA_READY Last status: IO error: INJECTED ERROR: tablet failed All reported replicas are: A = ce9933ca3a5c4f5ab9f3a41857b7fbb2 B = 7a66fc8acece469db093d36cf50b448e C = 2dd78402759d4877b2991ce5ad9abd06 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C* | 1 | -1 | Yes B | A B C* | 1 | -1 | Yes C | [config not available] | | | Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 0 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+---------------------+--------------- Small | 156e622c1244482eb132ee4e55767682 | 127.13.208.67:34059 | 0 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0917 09:19:58.890911 14145 cluster_verifier.cc:81] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0917 09:19:58.966661 16949 raft_consensus.cc:472] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2dd78402759d4877b2991ce5ad9abd06) I0917 09:19:58.966884 16949 raft_consensus.cc:494] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" member_type: VOTER last_known_addr { host: "127.13.208.68" port: 37535 } } peers { permanent_uuid: "7a66fc8acece469db093d36cf50b448e" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 46523 } } peers { permanent_uuid: "2dd78402759d4877b2991ce5ad9abd06" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39567 } } I0917 09:19:58.967012 16949 leader_election.cc:296] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a66fc8acece469db093d36cf50b448e (127.13.208.66:46523), 2dd78402759d4877b2991ce5ad9abd06 (127.13.208.65:39567) I0917 09:19:58.967509 16952 raft_consensus.cc:472] T 5e9d0599e0df416e97e63d608db6f0ab P 7a66fc8acece469db093d36cf50b448e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2dd78402759d4877b2991ce5ad9abd06) I0917 09:19:58.967839 16952 raft_consensus.cc:494] T 5e9d0599e0df416e97e63d608db6f0ab P 7a66fc8acece469db093d36cf50b448e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" member_type: VOTER last_known_addr { host: "127.13.208.68" port: 37535 } } peers { permanent_uuid: "7a66fc8acece469db093d36cf50b448e" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 46523 } } peers { permanent_uuid: "2dd78402759d4877b2991ce5ad9abd06" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39567 } } I0917 09:19:58.968256 16952 leader_election.cc:296] T 5e9d0599e0df416e97e63d608db6f0ab P 7a66fc8acece469db093d36cf50b448e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce9933ca3a5c4f5ab9f3a41857b7fbb2 (127.13.208.68:37535), 2dd78402759d4877b2991ce5ad9abd06 (127.13.208.65:39567) I0917 09:19:58.972733 16489 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5e9d0599e0df416e97e63d608db6f0ab" candidate_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2dd78402759d4877b2991ce5ad9abd06" is_pre_election: true I0917 09:19:58.972774 16557 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5e9d0599e0df416e97e63d608db6f0ab" candidate_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a66fc8acece469db093d36cf50b448e" is_pre_election: true I0917 09:19:58.972815 16557 raft_consensus.cc:2350] T 5e9d0599e0df416e97e63d608db6f0ab P 7a66fc8acece469db093d36cf50b448e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce9933ca3a5c4f5ab9f3a41857b7fbb2 in term 1. W0917 09:19:58.972956 16652 leader_election.cc:348] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2dd78402759d4877b2991ce5ad9abd06 (127.13.208.65:39567): Illegal state: cannot vote while shut down I0917 09:19:58.972985 16652 leader_election.cc:310] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a66fc8acece469db093d36cf50b448e, ce9933ca3a5c4f5ab9f3a41857b7fbb2; no voters: 2dd78402759d4877b2991ce5ad9abd06 I0917 09:19:58.973068 16949 raft_consensus.cc:2644] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0917 09:19:58.973085 16949 raft_consensus.cc:472] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2dd78402759d4877b2991ce5ad9abd06) I0917 09:19:58.973091 16949 raft_consensus.cc:2890] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [term 1 FOLLOWER]: Advancing to term 2 I0917 09:19:58.974056 16489 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5e9d0599e0df416e97e63d608db6f0ab" candidate_uuid: "7a66fc8acece469db093d36cf50b448e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2dd78402759d4877b2991ce5ad9abd06" is_pre_election: true I0917 09:19:58.974171 16691 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5e9d0599e0df416e97e63d608db6f0ab" candidate_uuid: "7a66fc8acece469db093d36cf50b448e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" is_pre_election: true W0917 09:19:58.974220 16518 leader_election.cc:348] T 5e9d0599e0df416e97e63d608db6f0ab P 7a66fc8acece469db093d36cf50b448e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2dd78402759d4877b2991ce5ad9abd06 (127.13.208.65:39567): Illegal state: cannot vote while shut down I0917 09:19:58.974772 16949 raft_consensus.cc:494] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" member_type: VOTER last_known_addr { host: "127.13.208.68" port: 37535 } } peers { permanent_uuid: "7a66fc8acece469db093d36cf50b448e" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 46523 } } peers { permanent_uuid: "2dd78402759d4877b2991ce5ad9abd06" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39567 } } I0917 09:19:58.974861 16691 raft_consensus.cc:2280] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a66fc8acece469db093d36cf50b448e in current term 2: Already voted for candidate ce9933ca3a5c4f5ab9f3a41857b7fbb2 in this term. I0917 09:19:58.974865 16949 leader_election.cc:296] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [CANDIDATE]: Term 2 election: Requested vote from peers 7a66fc8acece469db093d36cf50b448e (127.13.208.66:46523), 2dd78402759d4877b2991ce5ad9abd06 (127.13.208.65:39567) I0917 09:19:58.974994 16520 leader_election.cc:310] T 5e9d0599e0df416e97e63d608db6f0ab P 7a66fc8acece469db093d36cf50b448e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a66fc8acece469db093d36cf50b448e; no voters: 2dd78402759d4877b2991ce5ad9abd06, ce9933ca3a5c4f5ab9f3a41857b7fbb2 I0917 09:19:58.975092 16557 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5e9d0599e0df416e97e63d608db6f0ab" candidate_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7a66fc8acece469db093d36cf50b448e" I0917 09:19:58.975126 16952 raft_consensus.cc:2890] T 5e9d0599e0df416e97e63d608db6f0ab P 7a66fc8acece469db093d36cf50b448e [term 1 FOLLOWER]: Advancing to term 2 I0917 09:19:58.975148 16489 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5e9d0599e0df416e97e63d608db6f0ab" candidate_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2dd78402759d4877b2991ce5ad9abd06" W0917 09:19:58.975256 16652 leader_election.cc:348] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 2dd78402759d4877b2991ce5ad9abd06 (127.13.208.65:39567): Illegal state: cannot vote while shut down I0917 09:19:58.975775 16952 raft_consensus.cc:2592] T 5e9d0599e0df416e97e63d608db6f0ab P 7a66fc8acece469db093d36cf50b448e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0917 09:19:58.976397 16557 raft_consensus.cc:2350] T 5e9d0599e0df416e97e63d608db6f0ab P 7a66fc8acece469db093d36cf50b448e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce9933ca3a5c4f5ab9f3a41857b7fbb2 in term 2. I0917 09:19:58.976605 16655 leader_election.cc:310] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a66fc8acece469db093d36cf50b448e, ce9933ca3a5c4f5ab9f3a41857b7fbb2; no voters: 2dd78402759d4877b2991ce5ad9abd06 I0917 09:19:58.976734 16949 raft_consensus.cc:2644] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [term 2 FOLLOWER]: Leader election won for term 2 I0917 09:19:58.976851 16949 raft_consensus.cc:667] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [term 2 LEADER]: Becoming Leader. State: Replica: ce9933ca3a5c4f5ab9f3a41857b7fbb2, State: Running, Role: LEADER I0917 09:19:58.976965 16949 consensus_queue.cc:206] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" member_type: VOTER last_known_addr { host: "127.13.208.68" port: 37535 } } peers { permanent_uuid: "7a66fc8acece469db093d36cf50b448e" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 46523 } } peers { permanent_uuid: "2dd78402759d4877b2991ce5ad9abd06" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39567 } } I0917 09:19:58.977555 16405 catalog_manager.cc:4120] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 reported cstate change: term changed from 1 to 2, leader changed from 2dd78402759d4877b2991ce5ad9abd06 (127.13.208.65) to ce9933ca3a5c4f5ab9f3a41857b7fbb2 (127.13.208.68). New cstate: current_term: 2 leader_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" member_type: VOTER last_known_addr { host: "127.13.208.68" port: 37535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a66fc8acece469db093d36cf50b448e" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 46523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dd78402759d4877b2991ce5ad9abd06" member_type: VOTER last_known_addr { host: "127.13.208.65" port: 39567 } health_report { overall_health: UNKNOWN } } } I0917 09:19:59.575598 16557 raft_consensus.cc:1184] T 5e9d0599e0df416e97e63d608db6f0ab P 7a66fc8acece469db093d36cf50b448e [term 2 FOLLOWER]: Refusing update from remote peer ce9933ca3a5c4f5ab9f3a41857b7fbb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch) I0917 09:19:59.575964 16962 consensus_queue.cc:984] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a66fc8acece469db093d36cf50b448e" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 46523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0917 09:19:59.577363 16652 consensus_queue.cc:544] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [LEADER]: The tablet replica hosted on peer 2dd78402759d4877b2991ce5ad9abd06 has failed W0917 09:19:59.577416 16652 consensus_peers.cc:458] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 -> Peer 2dd78402759d4877b2991ce5ad9abd06 (127.13.208.65:39567): Couldn't send request to peer 2dd78402759d4877b2991ce5ad9abd06. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry. I0917 09:19:59.577702 16404 catalog_manager.cc:3871] Sending ChangeConfig:REMOVE_PEER on tablet 5e9d0599e0df416e97e63d608db6f0ab (attempt 1) I0917 09:19:59.577975 16691 consensus_queue.cc:206] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" member_type: VOTER last_known_addr { host: "127.13.208.68" port: 37535 } } peers { permanent_uuid: "7a66fc8acece469db093d36cf50b448e" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 46523 } } I0917 09:19:59.578204 16557 raft_consensus.cc:1184] T 5e9d0599e0df416e97e63d608db6f0ab P 7a66fc8acece469db093d36cf50b448e [term 2 FOLLOWER]: Refusing update from remote peer ce9933ca3a5c4f5ab9f3a41857b7fbb2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch) I0917 09:19:59.578457 16965 consensus_queue.cc:984] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a66fc8acece469db093d36cf50b448e" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 46523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0917 09:19:59.578997 16962 raft_consensus.cc:2792] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 2dd78402759d4877b2991ce5ad9abd06 (127.13.208.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" member_type: VOTER last_known_addr { host: "127.13.208.68" port: 37535 } } peers { permanent_uuid: "7a66fc8acece469db093d36cf50b448e" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 46523 } } } I0917 09:19:59.579165 16557 raft_consensus.cc:2792] T 5e9d0599e0df416e97e63d608db6f0ab P 7a66fc8acece469db093d36cf50b448e [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 2dd78402759d4877b2991ce5ad9abd06 (127.13.208.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" member_type: VOTER last_known_addr { host: "127.13.208.68" port: 37535 } } peers { permanent_uuid: "7a66fc8acece469db093d36cf50b448e" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 46523 } } } I0917 09:19:59.580726 16388 catalog_manager.cc:3654] ChangeConfig:REMOVE_PEER RPC for tablet 5e9d0599e0df416e97e63d608db6f0ab with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0917 09:19:59.581048 16404 catalog_manager.cc:4120] T 5e9d0599e0df416e97e63d608db6f0ab P 7a66fc8acece469db093d36cf50b448e reported cstate change: config changed from index -1 to 3, VOTER 2dd78402759d4877b2991ce5ad9abd06 (127.13.208.65) evicted. New cstate: current_term: 2 leader_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" member_type: VOTER last_known_addr { host: "127.13.208.68" port: 37535 } } peers { permanent_uuid: "7a66fc8acece469db093d36cf50b448e" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 46523 } } } I0917 09:19:59.581863 16404 catalog_manager.cc:3512] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 5e9d0599e0df416e97e63d608db6f0ab on 2dd78402759d4877b2991ce5ad9abd06 (127.13.208.65:39567) (TS 2dd78402759d4877b2991ce5ad9abd06 not found in new config with opid_index 3) I0917 09:19:59.581913 16405 catalog_manager.cc:3739] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 5e9d0599e0df416e97e63d608db6f0ab (attempt 1) I0917 09:19:59.582087 16473 tablet_service.cc:1066] Processing DeleteTablet for tablet 5e9d0599e0df416e97e63d608db6f0ab with delete_type TABLET_DATA_TOMBSTONED (TS 2dd78402759d4877b2991ce5ad9abd06 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54384 I0917 09:19:59.582260 16972 ts_tablet_manager.cc:1440] T 5e9d0599e0df416e97e63d608db6f0ab P 2dd78402759d4877b2991ce5ad9abd06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0917 09:19:59.582442 16405 catalog_manager.cc:3739] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 5e9d0599e0df416e97e63d608db6f0ab (attempt 1) I0917 09:19:59.582609 16691 consensus_queue.cc:206] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" member_type: VOTER last_known_addr { host: "127.13.208.68" port: 37535 } } peers { permanent_uuid: "7a66fc8acece469db093d36cf50b448e" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 46523 } } peers { permanent_uuid: "156e622c1244482eb132ee4e55767682" member_type: NON_VOTER last_known_addr { host: "127.13.208.67" port: 34059 } attrs { promote: true } } I0917 09:19:59.583142 16557 raft_consensus.cc:1184] T 5e9d0599e0df416e97e63d608db6f0ab P 7a66fc8acece469db093d36cf50b448e [term 2 FOLLOWER]: Refusing update from remote peer ce9933ca3a5c4f5ab9f3a41857b7fbb2: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch) I0917 09:19:59.583351 16965 consensus_queue.cc:984] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a66fc8acece469db093d36cf50b448e" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 46523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I0917 09:19:59.583390 16388 catalog_manager.cc:3671] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5e9d0599e0df416e97e63d608db6f0ab with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER failed with leader ce9933ca3a5c4f5ab9f3a41857b7fbb2 (127.13.208.68:37535) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time. Committed config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" member_type: VOTER last_known_addr { host: "127.13.208.68" port: 37535 } } peers { permanent_uuid: "7a66fc8acece469db093d36cf50b448e" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 46523 } }. Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" member_type: VOTER last_known_addr { host: "127.13.208.68" port: 37535 } } peers { permanent_uuid: "7a66fc8acece469db093d36cf50b448e" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 46523 } } peers { permanent_uuid: "156e622c1244482eb132ee4e55767682" member_type: NON_VOTER last_known_addr { host: "127.13.208.67" port: 34059 } attrs { promote: true } } I0917 09:19:59.583431 16388 catalog_manager.cc:3287] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5e9d0599e0df416e97e63d608db6f0ab with cas_config_opid_index 3 with a delay of 55 ms (attempt = 1) I0917 09:19:59.584429 16557 raft_consensus.cc:2792] T 5e9d0599e0df416e97e63d608db6f0ab P 7a66fc8acece469db093d36cf50b448e [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, NON_VOTER 156e622c1244482eb132ee4e55767682 (127.13.208.67) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" member_type: VOTER last_known_addr { host: "127.13.208.68" port: 37535 } } peers { permanent_uuid: "7a66fc8acece469db093d36cf50b448e" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 46523 } } peers { permanent_uuid: "156e622c1244482eb132ee4e55767682" member_type: NON_VOTER last_known_addr { host: "127.13.208.67" port: 34059 } attrs { promote: true } } } I0917 09:19:59.584468 16962 raft_consensus.cc:2792] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, NON_VOTER 156e622c1244482eb132ee4e55767682 (127.13.208.67) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" member_type: VOTER last_known_addr { host: "127.13.208.68" port: 37535 } } peers { permanent_uuid: "7a66fc8acece469db093d36cf50b448e" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 46523 } } peers { permanent_uuid: "156e622c1244482eb132ee4e55767682" member_type: NON_VOTER last_known_addr { host: "127.13.208.67" port: 34059 } attrs { promote: true } } } I0917 09:19:59.587106 16972 ts_tablet_manager.cc:1453] T 5e9d0599e0df416e97e63d608db6f0ab P 2dd78402759d4877b2991ce5ad9abd06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1 I0917 09:19:59.587153 16972 log.cc:990] T 5e9d0599e0df416e97e63d608db6f0ab P 2dd78402759d4877b2991ce5ad9abd06: Deleting WAL directory at /tmp/dist-test-taskLDE0O4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1568711978778636-14145/minicluster-data/ts-0-root/wals/5e9d0599e0df416e97e63d608db6f0ab I0917 09:19:59.587589 16388 catalog_manager.cc:3654] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5e9d0599e0df416e97e63d608db6f0ab with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0917 09:19:59.587632 16386 catalog_manager.cc:3490] TS 2dd78402759d4877b2991ce5ad9abd06 (127.13.208.65:39567): tablet 5e9d0599e0df416e97e63d608db6f0ab (table test-workload [id=1deae85f7ecc48d3ae00be11b1a14d9c]) successfully deleted I0917 09:19:59.587700 16405 catalog_manager.cc:4120] T 5e9d0599e0df416e97e63d608db6f0ab P 7a66fc8acece469db093d36cf50b448e reported cstate change: config changed from index 3 to 4, NON_VOTER 156e622c1244482eb132ee4e55767682 (127.13.208.67) added. New cstate: current_term: 2 leader_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ce9933ca3a5c4f5ab9f3a41857b7fbb2" member_type: VOTER last_known_addr { host: "127.13.208.68" port: 37535 } } peers { permanent_uuid: "7a66fc8acece469db093d36cf50b448e" member_type: VOTER last_known_addr { host: "127.13.208.66" port: 46523 } } peers { permanent_uuid: "156e622c1244482eb132ee4e55767682" member_type: NON_VOTER last_known_addr { host: "127.13.208.67" port: 34059 } attrs { promote: true } } } W0917 09:19:59.589591 16653 consensus_peers.cc:458] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 -> Peer 156e622c1244482eb132ee4e55767682 (127.13.208.67:34059): Couldn't send request to peer 156e622c1244482eb132ee4e55767682. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e9d0599e0df416e97e63d608db6f0ab. This is attempt 1: this message will repeat every 5th retry. I0917 09:19:59.639142 16388 catalog_manager.cc:3687] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5e9d0599e0df416e97e63d608db6f0ab with cas_config_opid_index 3: aborting the task: latest config opid_index 4; task opid_index 3 Master Summary UUID | Address | Status ----------------------------------+----------------------+--------- e3206a60b080419f84f498250a83fabf | 127.13.208.126:46829 | HEALTHY Flag | Value | Tags | Master ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+---------------------+---------+---------- 156e622c1244482eb132ee4e55767682 | 127.13.208.67:34059 | HEALTHY | 2dd78402759d4877b2991ce5ad9abd06 | 127.13.208.65:39567 | HEALTHY | 7a66fc8acece469db093d36cf50b448e | 127.13.208.66:46523 | HEALTHY | ce9933ca3a5c4f5ab9f3a41857b7fbb2 | 127.13.208.68:37535 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+-------+--------+------------------------- rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked Version Summary Version | Servers -----------------+------------------------- 1.11.0-SNAPSHOT | all 5 server(s) checked Tablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+---------+---------------+---------+------------+------------------+------------- test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 0 First Quartile | 1 Median | 1 Third Quartile | 1 Maximum | 1 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+---------------------+--------------- Small | 2dd78402759d4877b2991ce5ad9abd06 | 127.13.208.65:39567 | 0 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0917 09:19:59.679461 14145 log_verifier.cc:121] Checking tablet 5e9d0599e0df416e97e63d608db6f0ab I0917 09:19:59.685597 14145 log_verifier.cc:172] Verified matching terms for 4 ops in tablet 5e9d0599e0df416e97e63d608db6f0ab I0917 09:19:59.686223 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:59.690237 14145 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0917 09:19:59.701881 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:59.701915 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:59.701943 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:59.703025 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:59.706734 14145 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0917 09:19:59.706777 14145 tablet_replica.cc:273] T 5e9d0599e0df416e97e63d608db6f0ab P 7a66fc8acece469db093d36cf50b448e: stopping tablet replica I0917 09:19:59.706815 14145 raft_consensus.cc:2147] T 5e9d0599e0df416e97e63d608db6f0ab P 7a66fc8acece469db093d36cf50b448e [term 2 FOLLOWER]: Raft consensus shutting down. I0917 09:19:59.706847 14145 raft_consensus.cc:2174] T 5e9d0599e0df416e97e63d608db6f0ab P 7a66fc8acece469db093d36cf50b448e [term 2 FOLLOWER]: Raft consensus is shut down! I0917 09:19:59.719329 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:59.719365 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:59.719391 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:59.720160 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:59.723573 14145 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0917 09:19:59.734840 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:59.734875 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:59.734899 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:59.735600 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:59.738955 14145 ts_tablet_manager.cc:1159] Shutting down tablet manager... I0917 09:19:59.738986 14145 tablet_replica.cc:273] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2: stopping tablet replica I0917 09:19:59.739017 14145 raft_consensus.cc:2147] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [term 2 LEADER]: Raft consensus shutting down. I0917 09:19:59.739079 14145 raft_consensus.cc:2174] T 5e9d0599e0df416e97e63d608db6f0ab P ce9933ca3a5c4f5ab9f3a41857b7fbb2 [term 2 FOLLOWER]: Raft consensus is shut down! I0917 09:19:59.751610 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:59.751646 14145 tablet_server.cc:155] TabletServer shutting down... I0917 09:19:59.751669 14145 tablet_server.cc:169] TabletServer shutdown complete. I0917 09:19:59.752550 14145 master.cc:247] Master@127.13.208.126:46829 shutting down... I0917 09:19:59.755129 14145 raft_consensus.cc:2147] T 00000000000000000000000000000000 P e3206a60b080419f84f498250a83fabf [term 1 LEADER]: Raft consensus shutting down. I0917 09:19:59.755177 14145 raft_consensus.cc:2174] T 00000000000000000000000000000000 P e3206a60b080419f84f498250a83fabf [term 1 FOLLOWER]: Raft consensus is shut down! I0917 09:19:59.755188 14145 tablet_replica.cc:273] T 00000000000000000000000000000000 P e3206a60b080419f84f498250a83fabf: stopping tablet replica I0917 09:19:59.766757 14145 master.cc:258] Master@127.13.208.126:46829 shutdown complete. [ OK ] FailedTabletsAreReplacedITest.OneReplica/1 (3279 ms) [----------] 2 tests from FailedTabletsAreReplacedITest (7611 ms total) [----------] Global test environment tear-down [==========] 6 tests from 2 test cases ran. (20987 ms total) [ PASSED ] 5 tests. [ FAILED ] 1 test, listed below: [ FAILED ] TsTabletManagerITest.TestTableStats 1 FAILED TEST